[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:11 dist-test-slave-ph9j krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:11 dist-test-slave-ph9j krb5kdc[399](info): set up 2 sockets
Sep 24 23:06:11 dist-test-slave-ph9j krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 23:06:12.146045 392 mini_kdc.cc:121] Time spent starting KDC: real 2.680s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:12.264215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41669
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:12.343672 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:12.343904 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:12.343928 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:12.345091 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:12.345129 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:12.345142 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:12.345155 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:12.345167 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:12.346699 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:12.346906 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:12.347093 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:06:12.352970 412 server_base.cc:1034] running on GCE node
W20260924 23:06:12.353044 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:12.352936 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:12.352936 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:12.353482 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:12.353811 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:12.354995 412 hybrid_clock.cc:648] HybridClock initialized: now 1790291172354975 us; error 36 us; skew 500 ppm
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:12.360426 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:12.360965 412 webserver.cc:460] Webserver started at http://127.0.98.62:42713/ using document root <none> and password file <none>
I20260924 23:06:12.361212 412 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:12.361285 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:12.361447 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:12.362617 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "9e5f7d99e5d0468dac9e6d7908395fcc"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "9ec8eb98399de8351b595a89f2ba20ff"
server_key_iv: "cc0b3466c1edac53b0a4e15663ecf62a"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.363022 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e5f7d99e5d0468dac9e6d7908395fcc"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "9ec8eb98399de8351b595a89f2ba20ff"
server_key_iv: "cc0b3466c1edac53b0a4e15663ecf62a"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.364398 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:12.365212 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.365394 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:12.365497 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "9e5f7d99e5d0468dac9e6d7908395fcc"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "9ec8eb98399de8351b595a89f2ba20ff"
server_key_iv: "cc0b3466c1edac53b0a4e15663ecf62a"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.365590 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:12.379477 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:12.381618 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:12.381836 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:12.385656 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41669
I20260924 23:06:12.385708 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41669 every 8 connection(s)
I20260924 23:06:12.386154 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:12.387022 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:12.388904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 412
I20260924 23:06:12.389030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:12.389338 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc: Bootstrap starting.
I20260924 23:06:12.389731 392 external_mini_cluster.cc:1428] Setting key b4e2c1b213b7c21f317370a3d8900ad5
I20260924 23:06:12.389987 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:12.390283 480 log.cc:824] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:12.391096 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc: No bootstrap required, opened a new log
I20260924 23:06:12.392760 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260924 23:06:12.392926 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:12.392966 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5f7d99e5d0468dac9e6d7908395fcc, State: Initialized, Role: FOLLOWER
I20260924 23:06:12.393123 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260924 23:06:12.393201 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:12.393224 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:12.393286 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:12.394202 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260924 23:06:12.394289 480 leader_election.cc:302] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5f7d99e5d0468dac9e6d7908395fcc; no voters:
I20260924 23:06:12.394476 480 leader_election.cc:288] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:12.394536 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:12.394729 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [term 1 LEADER]: Becoming Leader. State: Replica: 9e5f7d99e5d0468dac9e6d7908395fcc, State: Running, Role: LEADER
I20260924 23:06:12.394747 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:12.394836 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } }
I20260924 23:06:12.395169 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e5f7d99e5d0468dac9e6d7908395fcc. Latest consensus state: current_term: 1 leader_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } } }
I20260924 23:06:12.395239 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:12.395190 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f7d99e5d0468dac9e6d7908395fcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41669 } } }
I20260924 23:06:12.395434 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:12.395464 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:12.396041 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:12.397606 491 catalog_manager.cc:1347] Generated new cluster ID: 94a8bc1ebe3b4aa1b4655f7ec614a019
I20260924 23:06:12.397655 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:12.407711 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.390707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55810 (local address 127.0.98.62:41669)
0924 23:06:12.390863 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:12.390873 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:12.391151 (+ 278us) server_negotiation.cc:407] Connection header received
0924 23:06:12.392803 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:12.392810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:12.392863 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:12.393019 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:12.394913 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.396334 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.397840 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.398469 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.398571 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.402915 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:12.402920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:12.402921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:12.402933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:12.405638 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.406143 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.406146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.406147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.406267 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.406558 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.406559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.406559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.406887 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:12.407048 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:12.407335 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 23:06:12.407479 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260924 23:06:12.411772 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:12.412278 491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:12.416688 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e5f7d99e5d0468dac9e6d7908395fcc: Generated new TSK 0
I20260924 23:06:12.416862 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:12.448932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:12.536868 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:12.537075 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:12.537099 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:12.538252 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:12.538293 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:12.538321 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:12.539708 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:12.539886 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:12.540066 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:12.544886 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:12.544956 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:12.544900 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:12.545095 506 server_base.cc:1034] running on GCE node
I20260924 23:06:12.545303 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:12.545499 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:12.546631 506 hybrid_clock.cc:648] HybridClock initialized: now 1790291172546619 us; error 29 us; skew 500 ppm
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:12.549032 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:12.549450 506 webserver.cc:460] Webserver started at http://127.0.98.1:35819/ using document root <none> and password file <none>
I20260924 23:06:12.549690 506 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:12.549739 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:12.549909 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:12.551049 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "b31d0319e6ca474183e8cbe7a2c0262d"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "bf0d4169628f857cf2dbfa094dd9aa50"
server_key_iv: "20881b7c5fd543f390b8d67a885274fb"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.551422 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b31d0319e6ca474183e8cbe7a2c0262d"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "bf0d4169628f857cf2dbfa094dd9aa50"
server_key_iv: "20881b7c5fd543f390b8d67a885274fb"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.552783 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:12.553557 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.553752 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:12.553844 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "b31d0319e6ca474183e8cbe7a2c0262d"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "bf0d4169628f857cf2dbfa094dd9aa50"
server_key_iv: "20881b7c5fd543f390b8d67a885274fb"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.553948 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:12.570230 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:12.572094 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:12.572254 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:12.572490 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:12.572815 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:12.572849 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.572872 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:12.572918 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.579056 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34833
I20260924 23:06:12.579106 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34833 every 8 connection(s)
I20260924 23:06:12.579437 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:12.583948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 506
I20260924 23:06:12.584072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:12.584234 392 external_mini_cluster.cc:1428] Setting key 95276b4348a5af56d8f1d02367f3807a
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:12.589708 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.580151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42395 (local address 127.0.98.62:41669)
0924 23:06:12.580230 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:12.580231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:12.580462 (+ 231us) server_negotiation.cc:407] Connection header received
0924 23:06:12.581014 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:12.581016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:12.581031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:12.581090 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:12.582113 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.582628 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.583661 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.583894 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.583976 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.586654 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:12.586656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:12.586656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:12.586664 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:12.588212 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.588681 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.588682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.588683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.588742 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.589007 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.589008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.589008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.589131 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:12.589210 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:12.589508 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 23:06:12.589603 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 23:06:12.589919 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.580152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41669 (local address 127.0.98.1:42395)
0924 23:06:12.580329 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:12.580366 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:12.580900 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:12.581165 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:12.581169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:12.581743 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:12.582013 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:12.582018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.582741 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.582743 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:12.583563 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:12.583564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.583983 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.583984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:12.584143 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.584756 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:12.584767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:12.586545 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:12.588332 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:12.588340 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:12.588342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:12.588581 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:12.588811 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:12.588812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:12.588812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:12.588935 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:12.589213 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:12.589220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:12.589368 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 23:06:12.589476 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 23:06:12.589642 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260924 23:06:12.590538 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41669
I20260924 23:06:12.590626 635 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:12.590837 635 heartbeater.cc:511] Master 127.0.98.62:41669 requested a full tablet report, sending...
I20260924 23:06:12.591434 444 ts_manager.cc:194] Registered new tserver with Master: b31d0319e6ca474183e8cbe7a2c0262d (127.0.98.1:34833)
I20260924 23:06:12.592434 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42395
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:12.612129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:12.693236 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:12.693447 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:12.693472 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:12.694778 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:12.694823 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:12.694851 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:12.696434 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:12.696696 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:12.696921 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:12.702176 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:12.702283 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:12.702421 640 server_base.cc:1034] running on GCE node
W20260924 23:06:12.702289 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:12.702692 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:12.703064 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:12.704245 640 hybrid_clock.cc:648] HybridClock initialized: now 1790291172704226 us; error 44 us; skew 500 ppm
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:12.706837 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:12.707327 640 webserver.cc:460] Webserver started at http://127.0.98.2:33651/ using document root <none> and password file <none>
I20260924 23:06:12.707552 640 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:12.707623 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:12.707777 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:12.708853 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "78dc885d35a845879078f6a53c6f6b54"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "f0928556666bfda4691a891cb064b698"
server_key_iv: "6eb75dfbb80af946ea08427dd0cbfdb6"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.709213 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78dc885d35a845879078f6a53c6f6b54"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "f0928556666bfda4691a891cb064b698"
server_key_iv: "6eb75dfbb80af946ea08427dd0cbfdb6"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.710682 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:12.711509 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.711680 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:12.711781 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "78dc885d35a845879078f6a53c6f6b54"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "f0928556666bfda4691a891cb064b698"
server_key_iv: "6eb75dfbb80af946ea08427dd0cbfdb6"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.711875 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:12.723631 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:12.725430 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:12.725590 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:12.725824 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:12.726158 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:12.726213 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.726263 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:12.726290 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.732440 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37977
I20260924 23:06:12.732481 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37977 every 8 connection(s)
I20260924 23:06:12.732802 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:12.736977 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 640
I20260924 23:06:12.737056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:12.737241 392 external_mini_cluster.cc:1428] Setting key dab8af7c4c41d78e4330a3369a4e9cb2
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:12.742702 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.733430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41669 (local address 127.0.98.2:42275)
0924 23:06:12.733597 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:12.733639 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:12.734199 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:12.734462 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:12.734466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:12.735044 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:12.735229 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:12.735234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.735938 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.735939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:12.736727 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:12.736729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.737150 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.737151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:12.737276 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.737841 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:12.737851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:12.739492 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:12.741297 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:12.741304 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:12.741306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:12.741581 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:12.741807 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:12.741808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:12.741808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:12.741959 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:12.742244 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:12.742249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:12.742364 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 23:06:12.742437 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 23:06:12.742551 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260924 23:06:12.742739 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.733460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42275 (local address 127.0.98.62:41669)
0924 23:06:12.733520 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:12.733522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:12.733722 (+ 200us) server_negotiation.cc:407] Connection header received
0924 23:06:12.734312 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:12.734314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:12.734341 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:12.734394 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:12.735340 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.735836 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.736849 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.737073 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.737142 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.739586 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:12.739588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:12.739588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:12.739596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:12.741208 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.741658 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.741659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.741660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.741724 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.742026 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.742028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.742028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.742174 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:12.742251 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:12.742510 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 23:06:12.742604 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:06:12.743134 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41669
I20260924 23:06:12.743238 769 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:12.743443 769 heartbeater.cc:511] Master 127.0.98.62:41669 requested a full tablet report, sending...
I20260924 23:06:12.743865 444 ts_manager.cc:194] Registered new tserver with Master: 78dc885d35a845879078f6a53c6f6b54 (127.0.98.2:37977)
I20260924 23:06:12.744230 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42275
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:12.764483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:12.848483 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:12.848680 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:12.848702 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:12.849898 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:12.849937 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:12.849964 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:12.851501 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41669
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:12.851688 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:12.851872 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:12.856828 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:12.856881 774 server_base.cc:1034] running on GCE node
W20260924 23:06:12.856912 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:12.856834 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:12.857203 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:12.857460 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:12.858620 774 hybrid_clock.cc:648] HybridClock initialized: now 1790291172858608 us; error 25 us; skew 500 ppm
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:12.861048 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:12.861483 774 webserver.cc:460] Webserver started at http://127.0.98.3:38067/ using document root <none> and password file <none>
I20260924 23:06:12.861717 774 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:12.861786 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:12.861953 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:12.863078 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "5cc82a78858c4310ab55fddf846846ed"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "fab99e3bd5532be3718842ea75152fa2"
server_key_iv: "49de0bb9a96ba6bf193a4dbda8a43917"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.863446 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5cc82a78858c4310ab55fddf846846ed"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "fab99e3bd5532be3718842ea75152fa2"
server_key_iv: "49de0bb9a96ba6bf193a4dbda8a43917"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.864779 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:12.865496 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.865669 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:12.865765 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "5cc82a78858c4310ab55fddf846846ed"
format_stamp: "Formatted at 2026-09-24 23:06:12 on dist-test-slave-ph9j"
server_key: "fab99e3bd5532be3718842ea75152fa2"
server_key_iv: "49de0bb9a96ba6bf193a4dbda8a43917"
server_key_version: "encryptionkey@0"
I20260924 23:06:12.865859 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:12.881191 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:12.883690 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:12.883870 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:12.884119 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:12.884455 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:12.884512 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.884546 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:12.884574 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:12.891346 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43057
I20260924 23:06:12.891387 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43057 every 8 connection(s)
I20260924 23:06:12.891717 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:12.899827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 774
I20260924 23:06:12.899942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:12.900097 392 external_mini_cluster.cc:1428] Setting key d093b411ff7901c95ba268c05f3f0588
I20260924 23:06:12.901473 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.892442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42639 (local address 127.0.98.62:41669)
0924 23:06:12.892505 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:12.892507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:12.892738 (+ 231us) server_negotiation.cc:407] Connection header received
0924 23:06:12.893218 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:12.893219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:12.893227 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:12.893285 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:12.894303 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.894795 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.895928 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.896193 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.896261 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.898394 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:12.898397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:12.898397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:12.898405 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:12.899860 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.900344 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.900345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.900345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.900432 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.900766 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.900767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.900767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.900906 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:12.900976 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:12.901306 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 23:06:12.901384 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:06:12.901608 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.892424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41669 (local address 127.0.98.3:42639)
0924 23:06:12.892589 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:12.892631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:12.893132 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:12.893348 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:12.893351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:12.893997 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:12.894210 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:12.894214 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.894880 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.894881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:12.895848 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:12.895850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.896286 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.896287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:12.896404 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.896887 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:12.896895 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:12.898285 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:12.899964 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:12.899971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:12.899973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:12.900260 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:12.900515 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:12.900516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:12.900516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:12.900683 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:12.900981 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:12.900986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:12.901144 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 23:06:12.901258 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:06:12.901406 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260924 23:06:12.902189 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41669
I20260924 23:06:12.902295 903 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:12.902467 903 heartbeater.cc:511] Master 127.0.98.62:41669 requested a full tablet report, sending...
I20260924 23:06:12.902843 444 ts_manager.cc:194] Registered new tserver with Master: 5cc82a78858c4310ab55fddf846846ed (127.0.98.3:43057)
I20260924 23:06:12.903219 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42639
I20260924 23:06:12.912106 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:12.934626 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.927272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.0.98.1:34833)
0924 23:06:12.927430 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:12.927434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:12.927447 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:06:12.927476 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:12.927478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:12.927492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:12.927558 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:12.928079 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.928808 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.929518 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.929838 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.929969 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.932122 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:12.932128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:12.932129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:12.932135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:12.933532 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.933834 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.933836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.933837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.933948 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.934124 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.934125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.934125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.934264 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:12.934364 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:12.934439 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 23:06:12.934500 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260924 23:06:12.935002 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55036
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:06:12 dist-test-slave-ph9j krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:12.956578 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:12.949714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.1:34833)
0924 23:06:12.949780 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:12.949784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:12.949918 (+ 134us) server_negotiation.cc:407] Connection header received
0924 23:06:12.950029 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:12.950030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:12.950041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:12.950082 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:12.950612 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.951236 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.951951 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:12.952147 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:12.952215 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:12.954282 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:12.954283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:12.954284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:12.954290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:12.955569 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.955894 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.955895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.955895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.955945 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:12.956139 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:12.956140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:12.956140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:12.956250 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:12.956346 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:12.956420 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:06:12.956479 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:06:12.957228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 506
I20260924 23:06:12.961719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 640
I20260924 23:06:12.966707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 774
I20260924 23:06:12.971045 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 412
2026-09-24T23:06:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3531 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:13 dist-test-slave-ph9j krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:13 dist-test-slave-ph9j krb5kdc[917](info): set up 2 sockets
Sep 24 23:06:13 dist-test-slave-ph9j krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 23:06:14.005453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:14.129303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46443
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:14.210346 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:14.210556 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:14.210578 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:14.211768 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:14.211817 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:14.211830 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:14.211843 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:14.211855 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:14.213331 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:14.213523 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:14.213773 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:14.218688 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.218806 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.218722 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:14.218925 930 server_base.cc:1034] running on GCE node
I20260924 23:06:14.219213 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:14.219481 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:14.220646 930 hybrid_clock.cc:648] HybridClock initialized: now 1790291174220623 us; error 35 us; skew 500 ppm
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:14.223390 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:14.223891 930 webserver.cc:460] Webserver started at http://127.0.98.62:36057/ using document root <none> and password file <none>
I20260924 23:06:14.224156 930 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:14.224208 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:14.224393 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:14.225555 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "fca9d0778a1c4a9ba5e58f98182895b8"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "8c79abfc35a279971cc86796da69c87d"
server_key_iv: "3f60ea74ae65e749f3461cb132428e3a"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.226028 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "fca9d0778a1c4a9ba5e58f98182895b8"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "8c79abfc35a279971cc86796da69c87d"
server_key_iv: "3f60ea74ae65e749f3461cb132428e3a"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.227500 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:14.228346 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.228508 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:14.228590 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "fca9d0778a1c4a9ba5e58f98182895b8"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "8c79abfc35a279971cc86796da69c87d"
server_key_iv: "3f60ea74ae65e749f3461cb132428e3a"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.228649 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:14.247763 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:14.250200 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:14.250313 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:14.254139 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46443
I20260924 23:06:14.254202 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46443 every 8 connection(s)
I20260924 23:06:14.254567 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:14.255465 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:14.257949 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8: Bootstrap starting.
I20260924 23:06:14.258562 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:14.258966 998 log.cc:824] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:14.259681 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8: No bootstrap required, opened a new log
I20260924 23:06:14.261270 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46443 } }
I20260924 23:06:14.261435 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:14.261482 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fca9d0778a1c4a9ba5e58f98182895b8, State: Initialized, Role: FOLLOWER
I20260924 23:06:14.261650 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46443 } }
I20260924 23:06:14.261754 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:14.261804 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:14.261857 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:14.262379 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46443 } }
I20260924 23:06:14.262501 998 leader_election.cc:302] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fca9d0778a1c4a9ba5e58f98182895b8; no voters:
I20260924 23:06:14.262670 998 leader_election.cc:288] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:14.262693 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:14.262936 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:14.263019 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [term 1 LEADER]: Becoming Leader. State: Replica: fca9d0778a1c4a9ba5e58f98182895b8, State: Running, Role: LEADER
I20260924 23:06:14.263105 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46443 } }
I20260924 23:06:14.263547 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46443 } } }
I20260924 23:06:14.263628 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:14.263827 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:14.264217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 930
I20260924 23:06:14.264305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:14.264453 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:14.264480 392 external_mini_cluster.cc:1428] Setting key a65381d61f8853bd36e24dbcf043e257
I20260924 23:06:14.263530 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fca9d0778a1c4a9ba5e58f98182895b8. Latest consensus state: current_term: 1 leader_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fca9d0778a1c4a9ba5e58f98182895b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46443 } } }
I20260924 23:06:14.264895 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:14.266168 1010 catalog_manager.cc:1347] Generated new cluster ID: d0f799f27f254a94b37f6a08508eb450
I20260924 23:06:14.266220 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:14.274345 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.265055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.0.98.62:46443)
0924 23:06:14.265203 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.265211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.265240 (+ 29us) server_negotiation.cc:407] Connection header received
0924 23:06:14.265364 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.265371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.265424 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.265572 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.266500 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.267396 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.268147 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.268565 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.268678 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.271048 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.271052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.271052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.271064 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.272772 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.273163 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.273166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.273166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.273261 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.273472 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.273473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.273473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.273729 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.273947 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.274029 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.274169 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 23:06:14.276563 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:14.277247 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:14.282374 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fca9d0778a1c4a9ba5e58f98182895b8: Generated new TSK 0
I20260924 23:06:14.282584 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:14.313362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46443
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:14.394605 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:14.394822 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:14.394855 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:14.396066 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:14.396117 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:14.396147 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:14.397696 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46443
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:14.397981 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:14.398250 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:14.403385 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.403393 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.403404 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:14.403860 1024 server_base.cc:1034] running on GCE node
I20260924 23:06:14.404094 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:14.404371 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:14.405580 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790291174405567 us; error 32 us; skew 500 ppm
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:14.408464 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:14.408932 1024 webserver.cc:460] Webserver started at http://127.0.98.1:33437/ using document root <none> and password file <none>
I20260924 23:06:14.409191 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:14.409264 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:14.409415 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:14.410566 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "b767ad81acf94ed694eb31dca066839d"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "92aede196245150c52dbb635389e3bb9"
server_key_iv: "040bd816b85fd26da71e7674e1185a32"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.410926 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b767ad81acf94ed694eb31dca066839d"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "92aede196245150c52dbb635389e3bb9"
server_key_iv: "040bd816b85fd26da71e7674e1185a32"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.412307 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:14.413132 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.413307 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:14.413408 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "b767ad81acf94ed694eb31dca066839d"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "92aede196245150c52dbb635389e3bb9"
server_key_iv: "040bd816b85fd26da71e7674e1185a32"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.413489 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:14.424643 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:14.427045 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:14.427218 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:14.427454 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:14.427795 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:14.427856 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.427893 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:14.427920 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.434036 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32769
I20260924 23:06:14.434089 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32769 every 8 connection(s)
I20260924 23:06:14.434406 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:14.438356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1024
I20260924 23:06:14.438475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:14.438642 392 external_mini_cluster.cc:1428] Setting key b884f433486f3f2678f19c1f12b41193
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:14.444638 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.435179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42737 (local address 127.0.98.62:46443)
0924 23:06:14.435271 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.435274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.435454 (+ 180us) server_negotiation.cc:407] Connection header received
0924 23:06:14.435953 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.435955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.435970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.436039 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.436952 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.437484 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.438675 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.438982 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.439060 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.441605 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.441607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.441608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.441616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.443199 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.443640 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.443641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.443642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.443708 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.443968 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.443969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.443969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.444093 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.444162 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.444454 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.444544 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 23:06:14.444842 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.435138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46443 (local address 127.0.98.1:42737)
0924 23:06:14.435322 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.435357 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.435847 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.436094 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.436096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.436648 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:14.436854 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.436859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.437585 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.437586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.438437 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.438440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.439069 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.439070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.439209 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.439790 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:14.439798 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:14.441499 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:14.443293 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:14.443300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:14.443303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:14.443548 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:14.443773 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:14.443774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:14.443775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:14.443891 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:14.444164 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:14.444170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:14.444314 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.444431 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.444594 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":110,"threads_started":1}
I20260924 23:06:14.445318 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46443
I20260924 23:06:14.445438 1153 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:14.445654 1153 heartbeater.cc:511] Master 127.0.98.62:46443 requested a full tablet report, sending...
I20260924 23:06:14.446215 962 ts_manager.cc:194] Registered new tserver with Master: b767ad81acf94ed694eb31dca066839d (127.0.98.1:32769)
I20260924 23:06:14.446846 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42737
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:14.469072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46443
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:14.554905 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:14.555107 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:14.555131 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:14.556265 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:14.556299 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:14.556325 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:14.557744 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46443
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:14.557993 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:14.558199 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:14.563285 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.563292 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:14.563580 1158 server_base.cc:1034] running on GCE node
W20260924 23:06:14.563301 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:14.563885 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:14.564129 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:14.565315 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790291174565318 us; error 51 us; skew 500 ppm
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:14.567744 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:14.568159 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46685/ using document root <none> and password file <none>
I20260924 23:06:14.568405 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:14.568473 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:14.568619 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:14.569646 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "a34ac20400df41e89951bcd4c4aa6c48"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "5f3b186a97c3240e7f40e3f12c55b3d4"
server_key_iv: "fd9ee421751dd041468aeed42a82fa35"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.570034 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a34ac20400df41e89951bcd4c4aa6c48"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "5f3b186a97c3240e7f40e3f12c55b3d4"
server_key_iv: "fd9ee421751dd041468aeed42a82fa35"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.571393 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:14.572142 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.572332 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:14.572430 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "a34ac20400df41e89951bcd4c4aa6c48"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "5f3b186a97c3240e7f40e3f12c55b3d4"
server_key_iv: "fd9ee421751dd041468aeed42a82fa35"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.572510 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:14.586506 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:14.588527 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:14.588696 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:14.588946 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:14.589260 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:14.589313 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.589381 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:14.589416 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.595155 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46351
I20260924 23:06:14.595281 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46351 every 8 connection(s)
I20260924 23:06:14.595559 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:14.601068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1158
I20260924 23:06:14.601186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:14.601357 392 external_mini_cluster.cc:1428] Setting key 75113240bde90e24556ac9db067f99fe
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:14.605721 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.596358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39131 (local address 127.0.98.62:46443)
0924 23:06:14.596432 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.596434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.596635 (+ 201us) server_negotiation.cc:407] Connection header received
0924 23:06:14.597143 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.597144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.597153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.597211 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.598230 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.598719 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.599805 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.600029 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.600087 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.602608 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.602610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.602611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.602619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.604164 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.604687 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.604688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.604689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.604751 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.604998 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.604999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.604999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.605147 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.605217 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.605483 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.605581 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:06:14.605965 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.596311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46443 (local address 127.0.98.2:39131)
0924 23:06:14.596488 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.596534 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.597057 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.597272 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.597276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.597927 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:14.598139 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.598144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.598805 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.598806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.599739 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.599741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.600110 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.600110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.600221 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.600829 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:14.600838 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:14.602507 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:14.604252 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:14.604260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:14.604263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:14.604610 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:14.604824 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:14.604825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:14.604825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:14.604934 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:14.605218 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:14.605222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:14.605348 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.605437 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.605612 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":117,"threads_started":1}
I20260924 23:06:14.606477 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46443
I20260924 23:06:14.606564 1287 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:14.606787 1287 heartbeater.cc:511] Master 127.0.98.62:46443 requested a full tablet report, sending...
I20260924 23:06:14.607203 962 ts_manager.cc:194] Registered new tserver with Master: a34ac20400df41e89951bcd4c4aa6c48 (127.0.98.2:46351)
I20260924 23:06:14.607741 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39131
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:14.626307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46443
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:14.706202 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:14.706404 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:14.706427 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:14.707587 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:14.707628 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:14.707655 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:14.709052 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46443
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:14.709280 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:14.709517 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:14.714311 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.714324 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:14.714406 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:14.714586 1292 server_base.cc:1034] running on GCE node
I20260924 23:06:14.714759 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:14.714943 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:14.716121 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790291174716110 us; error 28 us; skew 500 ppm
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:14.718739 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:14.719098 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46557/ using document root <none> and password file <none>
I20260924 23:06:14.719295 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:14.719334 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:14.719419 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:14.720435 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "a6f1dfb728854f82fc98ae4744146e80"
server_key_iv: "9f3ec80784aaa3480abe8da51750d2f1"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.720731 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "a6f1dfb728854f82fc98ae4744146e80"
server_key_iv: "9f3ec80784aaa3480abe8da51750d2f1"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.721997 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:14.722968 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.723196 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:14.723281 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf"
format_stamp: "Formatted at 2026-09-24 23:06:14 on dist-test-slave-ph9j"
server_key: "a6f1dfb728854f82fc98ae4744146e80"
server_key_iv: "9f3ec80784aaa3480abe8da51750d2f1"
server_key_version: "encryptionkey@0"
I20260924 23:06:14.723332 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:14.734253 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:14.736408 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:14.736516 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:14.736696 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:14.736989 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:14.737021 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.737042 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:14.737089 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:14.742844 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34811
I20260924 23:06:14.742892 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34811 every 8 connection(s)
I20260924 23:06:14.743182 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:14.750744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1292
I20260924 23:06:14.750859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:14.751037 392 external_mini_cluster.cc:1428] Setting key 8cdbf59d02af65a8d6b2846d6e3e44aa
I20260924 23:06:14.752563 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.743801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33897 (local address 127.0.98.62:46443)
0924 23:06:14.743847 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.743850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.744098 (+ 248us) server_negotiation.cc:407] Connection header received
0924 23:06:14.744541 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.744541 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.744551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.744604 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.745483 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.746027 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.747116 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.747290 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.747344 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.749555 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.749556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.749557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.749563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.751032 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.751579 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.751580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.751581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.751663 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.751960 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.751961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.751961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.752109 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.752167 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.752433 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.752501 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:14.752703 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.743805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46443 (local address 127.0.98.3:33897)
0924 23:06:14.743977 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.744011 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.744456 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.744680 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.744683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.745222 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:14.745405 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.745409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.746111 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.746112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.747035 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.747037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.747362 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.747363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.747462 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.748011 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:14.748019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:14.749466 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:14.751206 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:14.751212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:14.751214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:14.751499 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:14.751746 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:14.751747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:14.751747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:14.751886 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:14.752174 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:14.752179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:14.752290 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.752377 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.752493 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260924 23:06:14.753149 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46443
I20260924 23:06:14.753225 1421 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:14.753424 1421 heartbeater.cc:511] Master 127.0.98.62:46443 requested a full tablet report, sending...
I20260924 23:06:14.753786 962 ts_manager.cc:194] Registered new tserver with Master: 738f6d3b9c4a44cd9037aac35ff4b0cf (127.0.98.3:34811)
I20260924 23:06:14.754173 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33897
I20260924 23:06:14.762887 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:06:14.771368 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.764776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.0.98.62:46443)
0924 23:06:14.764842 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.764845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.764987 (+ 142us) server_negotiation.cc:407] Connection header received
0924 23:06:14.765040 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.765040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.765049 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.765097 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.765761 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.766349 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.767155 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.767352 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.767407 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.768804 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.768805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.768806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.768812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.770305 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.770619 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.770619 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.770620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.770674 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.770885 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.770885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.770885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.771016 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.771086 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.771198 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.771254 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:06:14.773507 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:14.775255 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:14.784227 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.780237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.0.98.1:32769)
0924 23:06:14.780726 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.780730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.780739 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:14.780777 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.780779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.780792 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.780871 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:14.780992 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.781676 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.782990 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.783747 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.783818 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.783886 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.784091 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.784130 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":116,"threads_started":1}
I20260924 23:06:14.784219 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.780130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32769 (local address 127.0.0.1:55304)
0924 23:06:14.780418 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.780427 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.780474 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.780881 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.780882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.780884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:14.780933 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.780935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.781798 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.781799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.782891 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.782894 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.783917 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.783918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.783980 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.783990 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.784075 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.784130 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":57,"threads_started":1}
I20260924 23:06:14.784582 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.778643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34811 (local address 127.0.0.1:33550)
0924 23:06:14.778792 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.778837 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.778903 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.780620 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.780622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.780625 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:14.780719 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.780721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.782526 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.782528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.783453 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.783455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.784338 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.784339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.784409 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.784413 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.784462 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.784504 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260924 23:06:14.784832 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.780255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33550 (local address 127.0.98.3:34811)
0924 23:06:14.780432 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.780439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.780451 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:14.780491 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.780493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.780511 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.780595 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:14.780803 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.782001 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.783555 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.784188 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.784256 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.784305 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.784703 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.784754 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260924 23:06:14.784999 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.779975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46351 (local address 127.0.0.1:44466)
0924 23:06:14.780282 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.780293 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.780347 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.780803 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.780804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.780806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:14.780849 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.780850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.782584 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.782585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.783784 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.783786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.784761 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.784762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.784823 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.784827 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.784886 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.784932 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":61,"threads_started":1}
I20260924 23:06:14.785159 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.780106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.0.98.2:46351)
0924 23:06:14.780588 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.780594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.780603 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:14.780639 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.780640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.780655 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.780730 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:14.780920 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.782024 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.783869 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.784597 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.784662 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.784729 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.784970 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.785016 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":89,"threads_started":1}
I20260924 23:06:14.785214 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb4c0114524c61ac6900a24ab4065b (DEFAULT_TABLE table=test-table [id=2752cf27738d4f12b00834c12dc7872c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:14.785462 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb4c0114524c61ac6900a24ab4065b (DEFAULT_TABLE table=test-table [id=2752cf27738d4f12b00834c12dc7872c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:14.785632 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb4c0114524c61ac6900a24ab4065b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:14.785835 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb4c0114524c61ac6900a24ab4065b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:14.785831 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb4c0114524c61ac6900a24ab4065b (DEFAULT_TABLE table=test-table [id=2752cf27738d4f12b00834c12dc7872c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:14.786144 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb4c0114524c61ac6900a24ab4065b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:14.788291 1435 tablet_bootstrap.cc:492] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf: Bootstrap starting.
I20260924 23:06:14.788861 1435 tablet_bootstrap.cc:654] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:14.789461 1435 log.cc:824] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:14.789605 1436 tablet_bootstrap.cc:492] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: Bootstrap starting.
I20260924 23:06:14.790091 1437 tablet_bootstrap.cc:492] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: Bootstrap starting.
I20260924 23:06:14.790294 1436 tablet_bootstrap.cc:654] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:14.790452 1435 tablet_bootstrap.cc:492] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf: No bootstrap required, opened a new log
I20260924 23:06:14.790526 1435 ts_tablet_manager.cc:1397] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:14.790796 1437 tablet_bootstrap.cc:654] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:14.790817 1436 log.cc:824] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:14.791345 1437 log.cc:824] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:14.791509 1436 tablet_bootstrap.cc:492] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: No bootstrap required, opened a new log
I20260924 23:06:14.791594 1436 ts_tablet_manager.cc:1397] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:06:14.792912 1436 raft_consensus.cc:348] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.792914 1435 raft_consensus.cc:348] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.793076 1435 raft_consensus.cc:374] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:14.793079 1436 raft_consensus.cc:374] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:14.793126 1436 raft_consensus.cc:729] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34ac20400df41e89951bcd4c4aa6c48, State: Initialized, Role: FOLLOWER
I20260924 23:06:14.793128 1435 raft_consensus.cc:729] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738f6d3b9c4a44cd9037aac35ff4b0cf, State: Initialized, Role: FOLLOWER
I20260924 23:06:14.793251 1436 consensus_queue.cc:260] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.793279 1435 consensus_queue.cc:260] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.793533 1287 heartbeater.cc:503] Master 127.0.98.62:46443 was elected leader, sending a full tablet report...
I20260924 23:06:14.793541 1436 ts_tablet_manager.cc:1428] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:06:14.793694 1435 ts_tablet_manager.cc:1428] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:14.793808 1421 heartbeater.cc:503] Master 127.0.98.62:46443 was elected leader, sending a full tablet report...
I20260924 23:06:14.794200 1437 tablet_bootstrap.cc:492] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: No bootstrap required, opened a new log
I20260924 23:06:14.794265 1437 ts_tablet_manager.cc:1397] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 23:06:14.795761 1437 raft_consensus.cc:348] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.795919 1437 raft_consensus.cc:374] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:14.795965 1437 raft_consensus.cc:729] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b767ad81acf94ed694eb31dca066839d, State: Initialized, Role: FOLLOWER
I20260924 23:06:14.796078 1437 consensus_queue.cc:260] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.796347 1437 ts_tablet_manager.cc:1428] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:06:14.796401 1153 heartbeater.cc:503] Master 127.0.98.62:46443 was elected leader, sending a full tablet report...
W20260924 23:06:14.846143 1288 tablet.cc:2369] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:14.872608 1441 raft_consensus.cc:482] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:14.872742 1441 raft_consensus.cc:504] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.873106 1441 leader_election.cc:288] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 738f6d3b9c4a44cd9037aac35ff4b0cf (127.0.98.3:34811), b767ad81acf94ed694eb31dca066839d (127.0.98.1:32769)
I20260924 23:06:14.876415 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.873441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43341 (local address 127.0.98.3:34811)
0924 23:06:14.873490 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.873493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.873501 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:14.873531 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.873532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.873545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.873603 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:14.873841 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.874498 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.875492 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.876086 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.876137 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.876145 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.876332 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.876362 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:14.876434 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.873258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34811 (local address 127.0.98.2:43341)
0924 23:06:14.873315 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.873328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.873395 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.873633 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.873635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.873637 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:14.873742 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.873744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.874585 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.874586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.875426 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.875429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.876172 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.876173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.876253 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.876258 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.876319 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.876358 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 23:06:14.876792 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb4c0114524c61ac6900a24ab4065b" candidate_uuid: "a34ac20400df41e89951bcd4c4aa6c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" is_pre_election: true
I20260924 23:06:14.876864 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.873442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34805 (local address 127.0.98.1:32769)
0924 23:06:14.873490 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.873492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.873608 (+ 116us) server_negotiation.cc:407] Connection header received
0924 23:06:14.873769 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.873770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.873783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.873842 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:14.874058 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.874676 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.875968 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.876536 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.876600 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.876611 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.876768 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.876810 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:06:14.876873 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.873343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32769 (local address 127.0.98.2:34805)
0924 23:06:14.873552 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:14.873561 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:14.873630 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:14.873895 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:14.873897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:14.873899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:14.873964 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.873966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.874777 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.874785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.875859 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:14.875862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.876627 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.876628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:14.876695 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.876698 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:14.876751 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:06:14.876786 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":110,"threads_started":1}
I20260924 23:06:14.876969 1375 raft_consensus.cc:2454] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34ac20400df41e89951bcd4c4aa6c48 in term 0.
I20260924 23:06:14.877204 1175 leader_election.cc:302] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 738f6d3b9c4a44cd9037aac35ff4b0cf, a34ac20400df41e89951bcd4c4aa6c48; no voters:
I20260924 23:06:14.877224 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb4c0114524c61ac6900a24ab4065b" candidate_uuid: "a34ac20400df41e89951bcd4c4aa6c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b767ad81acf94ed694eb31dca066839d" is_pre_election: true
I20260924 23:06:14.877365 1107 raft_consensus.cc:2454] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34ac20400df41e89951bcd4c4aa6c48 in term 0.
I20260924 23:06:14.877359 1441 raft_consensus.cc:2781] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:14.877429 1441 raft_consensus.cc:482] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:14.877465 1441 raft_consensus.cc:3037] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:14.878302 1441 raft_consensus.cc:504] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.878453 1441 leader_election.cc:288] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [CANDIDATE]: Term 1 election: Requested vote from peers 738f6d3b9c4a44cd9037aac35ff4b0cf (127.0.98.3:34811), b767ad81acf94ed694eb31dca066839d (127.0.98.1:32769)
I20260924 23:06:14.878590 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb4c0114524c61ac6900a24ab4065b" candidate_uuid: "a34ac20400df41e89951bcd4c4aa6c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b767ad81acf94ed694eb31dca066839d"
I20260924 23:06:14.878635 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb4c0114524c61ac6900a24ab4065b" candidate_uuid: "a34ac20400df41e89951bcd4c4aa6c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf"
I20260924 23:06:14.878679 1107 raft_consensus.cc:3037] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:14.878716 1375 raft_consensus.cc:3037] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:14.879412 1107 raft_consensus.cc:2454] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34ac20400df41e89951bcd4c4aa6c48 in term 1.
I20260924 23:06:14.879491 1375 raft_consensus.cc:2454] T 11cb4c0114524c61ac6900a24ab4065b P 738f6d3b9c4a44cd9037aac35ff4b0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34ac20400df41e89951bcd4c4aa6c48 in term 1.
I20260924 23:06:14.879582 1177 leader_election.cc:302] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a34ac20400df41e89951bcd4c4aa6c48, b767ad81acf94ed694eb31dca066839d; no voters:
I20260924 23:06:14.879680 1441 raft_consensus.cc:2781] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:14.879864 1441 raft_consensus.cc:686] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [term 1 LEADER]: Becoming Leader. State: Replica: a34ac20400df41e89951bcd4c4aa6c48, State: Running, Role: LEADER
I20260924 23:06:14.880115 1441 consensus_queue.cc:237] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } }
I20260924 23:06:14.881003 962 catalog_manager.cc:5495] T 11cb4c0114524c61ac6900a24ab4065b P a34ac20400df41e89951bcd4c4aa6c48 reported cstate change: term changed from 0 to 1, leader changed from <none> to a34ac20400df41e89951bcd4c4aa6c48 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a34ac20400df41e89951bcd4c4aa6c48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "738f6d3b9c4a44cd9037aac35ff4b0cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34ac20400df41e89951bcd4c4aa6c48" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b767ad81acf94ed694eb31dca066839d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32769 } health_report { overall_health: UNKNOWN } } }
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:14.901576 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.893958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.0.98.1:32769)
0924 23:06:14.894045 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.894048 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.894145 (+ 97us) server_negotiation.cc:407] Connection header received
0924 23:06:14.894251 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.894252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.894265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.894311 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.894883 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.895358 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.896070 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.896272 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.896356 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.898795 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.898806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.898806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.898813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.900485 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.900836 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.900839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.900839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.900894 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.901105 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.901105 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.901106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.901233 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.901332 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.901451 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.901510 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:14.925390 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.917941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55310 (local address 127.0.98.1:32769)
0924 23:06:14.918135 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.918137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.918219 (+ 82us) server_negotiation.cc:407] Connection header received
0924 23:06:14.918293 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.918294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.918304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.918389 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.918959 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.919486 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.920279 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.920476 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.920558 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.922895 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.922896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.922897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.922903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.924384 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.924711 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.924712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.924712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.924766 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.924932 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.924932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.924932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.925017 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.925084 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.925263 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.925323 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
W20260924 23:06:14.935135 1154 tablet.cc:2369] T 11cb4c0114524c61ac6900a24ab4065b P b767ad81acf94ed694eb31dca066839d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:06:14 dist-test-slave-ph9j krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:14.947674 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:14.940760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.0.98.1:32769)
0924 23:06:14.940833 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:14.940835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:14.940882 (+ 47us) server_negotiation.cc:407] Connection header received
0924 23:06:14.940927 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:14.940928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:14.940936 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:14.940984 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:14.941532 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.942070 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.942766 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:14.942941 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:14.942994 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:14.945252 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:14.945253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:14.945254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:14.945260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:14.946763 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.947072 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.947073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.947073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.947121 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:14.947281 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:14.947281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:14.947281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:14.947361 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:14.947426 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:14.947537 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 23:06:14.947586 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 23:06:14.948140 1067 tablet_service.cc:2908] Rejecting scan request for tablet 11cb4c0114524c61ac6900a24ab4065b: Uninitialized: clean time has not yet been initialized
I20260924 23:06:14.948923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1024
I20260924 23:06:14.953765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1158
I20260924 23:06:14.958443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1292
I20260924 23:06:14.962558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 930
2026-09-24T23:06:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1992 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:15 dist-test-slave-ph9j krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:15 dist-test-slave-ph9j krb5kdc[1459](info): set up 2 sockets
Sep 24 23:06:15 dist-test-slave-ph9j krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260924 23:06:15.997509 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:16 dist-test-slave-ph9j krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:16.117323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45683
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:16.197199 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:16.197448 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:16.197484 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:16.198637 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:16.198709 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 23:06:16.198729 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:16.198750 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:16.198777 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:16.198807 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:16.200276 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45683
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:16.200549 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:16.200764 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:16.205479 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:16.205520 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:16.205669 1472 server_base.cc:1034] running on GCE node
W20260924 23:06:16.205492 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:16.206035 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:16.206326 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:16.207491 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790291176207479 us; error 32 us; skew 500 ppm
Sep 24 23:06:16 dist-test-slave-ph9j krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:16.210101 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:16.210536 1472 webserver.cc:460] Webserver started at http://127.0.98.62:40033/ using document root <none> and password file <none>
I20260924 23:06:16.210773 1472 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:16.210842 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:16.210973 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:16.211968 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a"
format_stamp: "Formatted at 2026-09-24 23:06:16 on dist-test-slave-ph9j"
server_key: "fc86432a804d47fbaba4e31ad570059f"
server_key_iv: "61af2b27cf0d44d8665237db7e55aebe"
server_key_version: "encryptionkey@0"
I20260924 23:06:16.212324 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a"
format_stamp: "Formatted at 2026-09-24 23:06:16 on dist-test-slave-ph9j"
server_key: "fc86432a804d47fbaba4e31ad570059f"
server_key_iv: "61af2b27cf0d44d8665237db7e55aebe"
server_key_version: "encryptionkey@0"
I20260924 23:06:16.213613 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:16.214478 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:16.214651 1472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:16.214767 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a"
format_stamp: "Formatted at 2026-09-24 23:06:16 on dist-test-slave-ph9j"
server_key: "fc86432a804d47fbaba4e31ad570059f"
server_key_iv: "61af2b27cf0d44d8665237db7e55aebe"
server_key_version: "encryptionkey@0"
I20260924 23:06:16.214841 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:16.235891 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:16.238396 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:16.238547 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:16.243172 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45683
I20260924 23:06:16.243209 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45683 every 8 connection(s)
I20260924 23:06:16.243592 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:16.244263 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:16.247004 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: Bootstrap starting.
I20260924 23:06:16.247634 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:16.247936 1550 log.cc:824] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:16.248615 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: No bootstrap required, opened a new log
I20260924 23:06:16.250818 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45683 } }
I20260924 23:06:16.250981 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:16.251030 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ab5529e0e3a46d8b9f478a01d8a0d7a, State: Initialized, Role: FOLLOWER
I20260924 23:06:16.251195 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45683 } }
I20260924 23:06:16.251294 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:16.251348 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:16.251398 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:16.252318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1472
I20260924 23:06:16.252393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:16.252568 392 external_mini_cluster.cc:1428] Setting key d6ac6900aa676dd1818ec930ff5a2fb5
I20260924 23:06:16.253899 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45683 } }
I20260924 23:06:16.254014 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ab5529e0e3a46d8b9f478a01d8a0d7a; no voters:
I20260924 23:06:16.254170 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:16.254335 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:16.254431 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:16.254513 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [term 1 LEADER]: Becoming Leader. State: Replica: 3ab5529e0e3a46d8b9f478a01d8a0d7a, State: Running, Role: LEADER
I20260924 23:06:16.254601 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45683 } }
I20260924 23:06:16.254936 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45683 } } }
I20260924 23:06:16.255005 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:16.255111 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ab5529e0e3a46d8b9f478a01d8a0d7a. Latest consensus state: current_term: 1 leader_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ab5529e0e3a46d8b9f478a01d8a0d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45683 } } }
I20260924 23:06:16.255160 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:16.255280 1563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:16.255745 1563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:16.258687 1572 mvcc.cc:205] Tried to move back new op lower bound from 7333032657947561984 to 7333032657939189760. Current Snapshot: MvccSnapshot[applied={T|T < 7333032657947561984}]
I20260924 23:06:16.259788 1563 catalog_manager.cc:1347] Generated new cluster ID: 00b837bfedf94d1bbd58320d0f89f275
I20260924 23:06:16.259851 1563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:16 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:16.265384 1554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:16.253096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34488 (local address 127.0.98.62:45683)
0924 23:06:16.253272 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:16.253289 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:16.253314 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:06:16.253407 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:16.253412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:16.253452 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:16.253560 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:16.254650 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.256172 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.257117 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.257804 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.257959 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:16.260814 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:16.260818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:16.260819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:16.260832 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:16.263580 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:16.264016 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:16.264019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:16.264019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:16.264200 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:16.264434 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:16.264435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:16.264435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:16.264723 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:16.264867 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:16.265014 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 23:06:16.265151 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260924 23:06:16.278179 1563 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:16.278770 1563 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:16.286931 1563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: Generated new TSK 0
I20260924 23:06:16.287221 1563 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260924 23:06:16.770779 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.475s user 0.001s sys 0.002s
I20260924 23:06:16.771118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45683
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:16.855620 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:16.855819 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:16.855842 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:16.857093 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:16.857139 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:16.857160 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:16.857218 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:16.858760 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:16.859007 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:16.859236 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:16.864185 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:16.864239 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:16.864185 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:16.864531 1576 server_base.cc:1034] running on GCE node
I20260924 23:06:16.864730 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:16.864984 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:16.866122 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790291176866101 us; error 37 us; skew 500 ppm
Sep 24 23:06:16 dist-test-slave-ph9j krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:16.868510 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:16.868983 1576 webserver.cc:460] Webserver started at http://127.0.98.1:44131/ using document root <none> and password file <none>
I20260924 23:06:16.869204 1576 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:16.869249 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:16.869415 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:16.870507 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4"
format_stamp: "Formatted at 2026-09-24 23:06:16 on dist-test-slave-ph9j"
server_key: "4b42e1d15b63cd3121c788371f879d42"
server_key_iv: "95798217d933f79b7ea5b54d9ea79cc6"
server_key_version: "encryptionkey@0"
I20260924 23:06:16.870863 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4"
format_stamp: "Formatted at 2026-09-24 23:06:16 on dist-test-slave-ph9j"
server_key: "4b42e1d15b63cd3121c788371f879d42"
server_key_iv: "95798217d933f79b7ea5b54d9ea79cc6"
server_key_version: "encryptionkey@0"
I20260924 23:06:16.872167 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:16.872867 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:16.873060 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:16.873150 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4"
format_stamp: "Formatted at 2026-09-24 23:06:16 on dist-test-slave-ph9j"
server_key: "4b42e1d15b63cd3121c788371f879d42"
server_key_iv: "95798217d933f79b7ea5b54d9ea79cc6"
server_key_version: "encryptionkey@0"
I20260924 23:06:16.873204 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:16.894982 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:16.896973 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:16.897085 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:16.897545 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:16.897581 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:16.897604 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:16.897616 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:06:16 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:16.906635 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43175
I20260924 23:06:16.906752 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43175 every 8 connection(s)
I20260924 23:06:16.907065 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:16.910156 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:16.898182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50663 (local address 127.0.98.62:45683)
0924 23:06:16.898307 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:16.898310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:16.898487 (+ 177us) server_negotiation.cc:407] Connection header received
0924 23:06:16.899077 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:16.899079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:16.899095 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:16.899182 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:16.900100 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.900995 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.902745 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.903077 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.903150 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:16.906311 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:16.906313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:16.906313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:16.906323 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:16.908542 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:16.908979 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:16.908980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:16.908981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:16.909045 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:16.909340 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:16.909341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:16.909341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:16.909523 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:16.909594 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:16.909946 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 23:06:16.910038 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260924 23:06:16.910393 1601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:16.898165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.98.1:50663)
0924 23:06:16.898344 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:16.898406 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:16.898920 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:16.899277 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:16.899283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:16.899670 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:16.899992 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:16.899998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.901117 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.901118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:16.902637 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:16.902641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.903228 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.903229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:16.903416 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:16.904060 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:16.904071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:16.906192 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:16.908628 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:16.908635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:16.908637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:16.908914 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:16.909123 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:16.909124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:16.909124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:16.909271 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:16.909585 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:16.909592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:16.909757 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 23:06:16.909901 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 23:06:16.910120 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260924 23:06:16.910918 1708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45683
I20260924 23:06:16.911001 1708 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:16.911182 1708 heartbeater.cc:511] Master 127.0.98.62:45683 requested a full tablet report, sending...
I20260924 23:06:16.911685 1503 ts_manager.cc:194] Registered new tserver with Master: b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175)
I20260924 23:06:16.912544 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50663
I20260924 23:06:16.916065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1576
I20260924 23:06:16.916167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:16.916366 392 external_mini_cluster.cc:1428] Setting key 6168cbfb7149e71b0beda21d35adb768
I20260924 23:06:16.917436 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:16.911992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33279 (local address 127.0.98.62:45683)
0924 23:06:16.912030 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:16.912032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:16.912103 (+ 71us) server_negotiation.cc:407] Connection header received
0924 23:06:16.912169 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:16.912170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:16.912179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:16.912234 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:16.912499 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.913152 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.913939 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.914139 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.914190 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:16.914982 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:16.914983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:16.914983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:16.914989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:16.916275 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:16.916632 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:16.916633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:16.916633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:16.916693 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:16.916914 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:16.916915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:16.916915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:16.917033 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:16.917093 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:16.917270 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 23:06:16.917345 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:16.917443 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:16.911933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.98.1:33279)
0924 23:06:16.912047 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:16.912058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:16.912116 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:16.912236 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:16.912236 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:16.912376 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:16.912436 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:16.912437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.913249 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.913250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:16.913782 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:16.913784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:16.914200 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:16.914200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:16.914249 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:16.914605 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:16.914609 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:16.914910 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:16.916354 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:16.916354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:16.916355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:16.916557 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:16.916767 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:16.916769 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:16.916769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:16.916838 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:16.917099 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:16.917100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:16.917151 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 23:06:16.917233 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:06:16.917338 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:16.941283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45683
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:17.022153 1717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:17.022352 1717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:17.022377 1717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:17.023470 1717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:17.023505 1717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:17.023526 1717 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:17.023543 1717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:17.024971 1717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:17.025207 1717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:17.025418 1717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:17.030242 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:17.030360 1717 server_base.cc:1034] running on GCE node
W20260924 23:06:17.030256 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:17.030315 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:17.030625 1717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:17.030893 1717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:17.032068 1717 hybrid_clock.cc:648] HybridClock initialized: now 1790291177032053 us; error 26 us; skew 500 ppm
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:17.038112 1717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:17.038523 1717 webserver.cc:460] Webserver started at http://127.0.98.2:46525/ using document root <none> and password file <none>
I20260924 23:06:17.038720 1717 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:17.038761 1717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:17.038851 1717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:17.039860 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "e2c9ee48c546407eb135fd383657fc63"
format_stamp: "Formatted at 2026-09-24 23:06:17 on dist-test-slave-ph9j"
server_key: "b0d02d648e5cbfe9bb0dab9201e1d262"
server_key_iv: "c5311034aac157ccf9361c06888495fc"
server_key_version: "encryptionkey@0"
I20260924 23:06:17.040169 1717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e2c9ee48c546407eb135fd383657fc63"
format_stamp: "Formatted at 2026-09-24 23:06:17 on dist-test-slave-ph9j"
server_key: "b0d02d648e5cbfe9bb0dab9201e1d262"
server_key_iv: "c5311034aac157ccf9361c06888495fc"
server_key_version: "encryptionkey@0"
I20260924 23:06:17.041390 1717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:17.042141 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.042291 1717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:17.042363 1717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "e2c9ee48c546407eb135fd383657fc63"
format_stamp: "Formatted at 2026-09-24 23:06:17 on dist-test-slave-ph9j"
server_key: "b0d02d648e5cbfe9bb0dab9201e1d262"
server_key_iv: "c5311034aac157ccf9361c06888495fc"
server_key_version: "encryptionkey@0"
I20260924 23:06:17.042416 1717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:17.052791 1717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:17.054951 1717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:17.055104 1717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:17.055685 1717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:17.055740 1717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.055765 1717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:17.055778 1717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.062189 1717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46549
I20260924 23:06:17.062367 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46549 every 8 connection(s)
I20260924 23:06:17.062623 1717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:17.065788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1717
I20260924 23:06:17.065917 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:17.066057 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.056119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53373 (local address 127.0.98.62:45683)
0924 23:06:17.056191 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.056193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.056447 (+ 254us) server_negotiation.cc:407] Connection header received
0924 23:06:17.056994 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.056996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.057010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.057090 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.057944 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.058785 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.060359 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.060712 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.060792 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.063676 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.063677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.063677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.063683 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.064764 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.065113 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.065114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.065114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.065163 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.065357 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.065357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.065357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.065457 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.065505 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.065823 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.065952 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:06:17.066110 392 external_mini_cluster.cc:1428] Setting key 9afa074ea47695c3912781b82bcbf848
I20260924 23:06:17.066231 1741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.056095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.98.2:53373)
0924 23:06:17.056343 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.056371 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.056852 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.057170 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.057174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.057529 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:17.057824 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.057831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.058888 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.058889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.060263 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.060265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.060864 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.060866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.061031 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.061661 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:17.061670 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:17.063596 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:17.064833 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.064838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.064840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.065047 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.065204 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.065204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.065204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.065301 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.065514 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:17.065518 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:17.065675 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.065806 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.066002 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":95,"threads_started":1}
I20260924 23:06:17.066776 1848 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45683
I20260924 23:06:17.066864 1848 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:17.067045 1848 heartbeater.cc:511] Master 127.0.98.62:45683 requested a full tablet report, sending...
I20260924 23:06:17.067425 1504 ts_manager.cc:194] Registered new tserver with Master: e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549)
I20260924 23:06:17.067814 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53373
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 23:06:17.074167 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.068508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40485 (local address 127.0.98.62:45683)
0924 23:06:17.068549 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.068551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.068626 (+ 75us) server_negotiation.cc:407] Connection header received
0924 23:06:17.068694 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.068695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.068705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.068760 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.069065 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.069546 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.070398 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.070595 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.070664 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.071685 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.071686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.071686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.071693 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.072961 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.073297 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.073298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.073298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.073363 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.073566 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.073567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.073568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.073719 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.073796 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.073998 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.074073 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:17.074179 1855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.068436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.98.2:40485)
0924 23:06:17.068565 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.068577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.068637 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.068765 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.068765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.068935 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:17.068987 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.068988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.069640 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.069641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.070305 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.070307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.070675 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.070676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.070738 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.071258 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:17.071262 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:17.071595 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:17.073049 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.073050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.073051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.073223 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.073437 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.073438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.073438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.073503 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.073798 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:17.073799 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:17.073850 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.073972 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.074073 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:17.094027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45683
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:17.182487 1857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:17.182807 1857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:17.182862 1857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:17.185045 1857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:17.185112 1857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:17.185150 1857 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:17.185189 1857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:17.187423 1857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:17.187717 1857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:17.188031 1857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:17.195683 1865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:17.195683 1863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:17.195973 1862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:17.195947 1857 server_base.cc:1034] running on GCE node
I20260924 23:06:17.196265 1857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:17.196559 1857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:17.197743 1857 hybrid_clock.cc:648] HybridClock initialized: now 1790291177197717 us; error 34 us; skew 500 ppm
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:17.201455 1857 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:17.202006 1857 webserver.cc:460] Webserver started at http://127.0.98.3:34291/ using document root <none> and password file <none>
I20260924 23:06:17.202299 1857 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:17.202378 1857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:17.202533 1857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:17.204049 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "414c8cf789e1440082e81be44e33241a"
format_stamp: "Formatted at 2026-09-24 23:06:17 on dist-test-slave-ph9j"
server_key: "e65d45ee04f1927f7696a917eaee048d"
server_key_iv: "8daacceddcaf774ea9373561945d2166"
server_key_version: "encryptionkey@0"
I20260924 23:06:17.204512 1857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "414c8cf789e1440082e81be44e33241a"
format_stamp: "Formatted at 2026-09-24 23:06:17 on dist-test-slave-ph9j"
server_key: "e65d45ee04f1927f7696a917eaee048d"
server_key_iv: "8daacceddcaf774ea9373561945d2166"
server_key_version: "encryptionkey@0"
I20260924 23:06:17.206357 1857 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260924 23:06:17.207415 1872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.207625 1857 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:17.207736 1857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "414c8cf789e1440082e81be44e33241a"
format_stamp: "Formatted at 2026-09-24 23:06:17 on dist-test-slave-ph9j"
server_key: "e65d45ee04f1927f7696a917eaee048d"
server_key_iv: "8daacceddcaf774ea9373561945d2166"
server_key_version: "encryptionkey@0"
I20260924 23:06:17.207831 1857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:17.255008 1857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:17.257987 1857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:17.258152 1857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:17.258677 1857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:17.258746 1857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.258791 1857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:17.258817 1857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.265733 1857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43035
I20260924 23:06:17.265803 1987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43035 every 8 connection(s)
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:17.266171 1857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:17.269529 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.259169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42239 (local address 127.0.98.62:45683)
0924 23:06:17.259231 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.259233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.259468 (+ 235us) server_negotiation.cc:407] Connection header received
0924 23:06:17.260002 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.260004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.260015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.260077 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.261067 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.261854 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.263437 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.263779 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.263875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.266537 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.266538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.266539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.266547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.268090 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.268505 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.268506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.268507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.268564 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.268859 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.268860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.268861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.268992 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.269060 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.269343 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.269419 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:06:17.269680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 1857
I20260924 23:06:17.269716 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.259204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.98.3:42239)
0924 23:06:17.259341 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.259400 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.259897 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.260201 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.260207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.260605 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:17.260972 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.260979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.261973 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.261974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.263338 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.263341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.263871 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.263872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.264004 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.264504 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:17.264514 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:17.266444 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:17.268190 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.268197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.268199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.268427 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.268667 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.268668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.268668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.268784 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.269062 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:17.269068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:17.269206 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.269321 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.269479 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260924 23:06:17.269752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:17.270035 392 external_mini_cluster.cc:1428] Setting key cc776fc42edbb8555cbc833dc0c42ea7
I20260924 23:06:17.270370 1988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45683
I20260924 23:06:17.270475 1988 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:17.270653 1988 heartbeater.cc:511] Master 127.0.98.62:45683 requested a full tablet report, sending...
I20260924 23:06:17.271030 1503 ts_manager.cc:194] Registered new tserver with Master: 414c8cf789e1440082e81be44e33241a (127.0.98.3:43035)
I20260924 23:06:17.271525 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42239
I20260924 23:06:17.276964 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.271454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37513 (local address 127.0.98.62:45683)
0924 23:06:17.271498 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.271500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.271623 (+ 123us) server_negotiation.cc:407] Connection header received
0924 23:06:17.271693 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.271694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.271704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.271764 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.272082 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.272544 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.273236 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.273443 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.273512 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.274479 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.274481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.274481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.274488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.275859 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.276196 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.276197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.276198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.276249 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.276476 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.276476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.276477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.276581 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.276652 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.276807 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.276871 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:17.276978 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.271403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.98.3:37513)
0924 23:06:17.271549 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.271561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.271636 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.271770 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.271771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.271948 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:17.271999 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.272001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.272631 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.272631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.273151 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.273152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.273521 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.273522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.273590 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.274081 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:17.274086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:17.274376 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:17.275955 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.275956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.275957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.276123 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.276337 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.276338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.276338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.276402 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.276644 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:17.276645 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:17.276687 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.276771 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.276873 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260924 23:06:17.281507 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:17.304342 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.297107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.0.98.62:45683)
0924 23:06:17.297171 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.297172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.297226 (+ 54us) server_negotiation.cc:407] Connection header received
0924 23:06:17.297287 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.297288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.297297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.297341 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.297990 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.298543 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.299281 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.299481 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.299542 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.301714 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.301716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.301721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.301729 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.303385 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.303694 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.303695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.303695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.303757 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.303947 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.303948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.303948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.304055 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.304134 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.304214 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.304273 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:06:17.306238 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:17.307891 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:17.314736 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.310517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43035 (local address 127.0.0.1:50608)
0924 23:06:17.310741 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.310786 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.310865 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.311320 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.311324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.311326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.311405 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.311407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.312292 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.312293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.313499 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.313501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.314430 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.314431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.314506 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.314522 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.314595 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.314646 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":91,"threads_started":1}
I20260924 23:06:17.314762 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.310614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.0.98.3:43035)
0924 23:06:17.310796 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.310800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.310864 (+ 64us) server_negotiation.cc:407] Connection header received
0924 23:06:17.310956 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.310958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.310976 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.311064 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.311502 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.312201 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.313593 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.314341 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.314421 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.314493 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.314617 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.314652 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":81,"threads_started":1}
I20260924 23:06:17.315357 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.310517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43175 (local address 127.0.0.1:36860)
0924 23:06:17.311138 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.311148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.311224 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.311379 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.311380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.311381 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.311408 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.311409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.312706 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.312707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.314073 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.314076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.315086 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.315088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.315156 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.315160 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.315228 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.315280 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":7,"thread_start_us":93,"threads_started":1}
I20260924 23:06:17.315555 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.310965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.0.98.2:46549)
0924 23:06:17.311188 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.311203 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.311214 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:17.311253 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.311255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.311271 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.311348 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.311555 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.312619 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.313831 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.314529 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.314604 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.314677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.315398 (+ 721us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.315442 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260924 23:06:17.315358 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.310761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46549 (local address 127.0.0.1:35752)
0924 23:06:17.310980 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.310989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.311031 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.311420 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.311421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.311424 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.311478 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.311480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.312775 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.312776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.313736 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.313738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.315153 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.315153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.315208 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.315212 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.315259 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.315292 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260924 23:06:17.315692 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.310626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36860 (local address 127.0.98.1:43175)
0924 23:06:17.310927 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.310931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.311224 (+ 293us) server_negotiation.cc:407] Connection header received
0924 23:06:17.311294 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.311297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.311312 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.311416 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.311478 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.312589 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.314164 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.314909 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.314982 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.315054 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.315549 (+ 495us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.315596 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":73,"threads_started":1}
I20260924 23:06:17.315600 1914 tablet_service.cc:1467] Processing CreateTablet for tablet c6076bdde6bf4c0388a291fd14c282ba (DEFAULT_TABLE table=test-table [id=2f0dd8cf2cb64ddc923ccd5df102fa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:17.316005 1914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6076bdde6bf4c0388a291fd14c282ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.316244 1642 tablet_service.cc:1467] Processing CreateTablet for tablet c6076bdde6bf4c0388a291fd14c282ba (DEFAULT_TABLE table=test-table [id=2f0dd8cf2cb64ddc923ccd5df102fa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:17.316239 1782 tablet_service.cc:1467] Processing CreateTablet for tablet c6076bdde6bf4c0388a291fd14c282ba (DEFAULT_TABLE table=test-table [id=2f0dd8cf2cb64ddc923ccd5df102fa3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:17.316536 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6076bdde6bf4c0388a291fd14c282ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.316536 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6076bdde6bf4c0388a291fd14c282ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.319132 2012 tablet_bootstrap.cc:492] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Bootstrap starting.
I20260924 23:06:17.319824 2012 tablet_bootstrap.cc:654] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.320112 2012 log.cc:824] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:17.320499 2013 tablet_bootstrap.cc:492] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Bootstrap starting.
I20260924 23:06:17.320861 2012 tablet_bootstrap.cc:492] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: No bootstrap required, opened a new log
I20260924 23:06:17.320916 2012 ts_tablet_manager.cc:1397] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:17.321074 2013 tablet_bootstrap.cc:654] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.321363 2013 log.cc:824] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:17.321820 2014 tablet_bootstrap.cc:492] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Bootstrap starting.
I20260924 23:06:17.322319 2014 tablet_bootstrap.cc:654] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.322605 2014 log.cc:824] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:17.322772 2012 raft_consensus.cc:348] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.322924 2012 raft_consensus.cc:374] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.322966 2012 raft_consensus.cc:729] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414c8cf789e1440082e81be44e33241a, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.323078 2012 consensus_queue.cc:260] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.323167 2013 tablet_bootstrap.cc:492] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: No bootstrap required, opened a new log
I20260924 23:06:17.323246 2013 ts_tablet_manager.cc:1397] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:06:17.323351 2012 ts_tablet_manager.cc:1428] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 23:06:17.323375 1988 heartbeater.cc:503] Master 127.0.98.62:45683 was elected leader, sending a full tablet report...
I20260924 23:06:17.324936 2013 raft_consensus.cc:348] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.325063 2013 raft_consensus.cc:374] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.325084 2013 raft_consensus.cc:729] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c9ee48c546407eb135fd383657fc63, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.325196 2013 consensus_queue.cc:260] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.325409 2013 ts_tablet_manager.cc:1428] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:17.325477 1848 heartbeater.cc:503] Master 127.0.98.62:45683 was elected leader, sending a full tablet report...
I20260924 23:06:17.325855 2014 tablet_bootstrap.cc:492] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: No bootstrap required, opened a new log
I20260924 23:06:17.325943 2014 ts_tablet_manager.cc:1397] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 23:06:17.327443 2014 raft_consensus.cc:348] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.327570 2014 raft_consensus.cc:374] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.327591 2014 raft_consensus.cc:729] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b076e0eb2b6d4e8b85a1a04e6be213c4, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.327699 2014 consensus_queue.cc:260] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.327916 2014 ts_tablet_manager.cc:1428] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:06:17.327987 1708 heartbeater.cc:503] Master 127.0.98.62:45683 was elected leader, sending a full tablet report...
I20260924 23:06:17.388466 2018 raft_consensus.cc:482] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:17.388597 2018 raft_consensus.cc:504] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.389019 2018 leader_election.cc:288] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175), e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549)
I20260924 23:06:17.392045 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.389231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43175 (local address 127.0.98.3:55343)
0924 23:06:17.389281 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.389295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.389341 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.389545 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.389547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.389549 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.389622 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.389626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.390428 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.390429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.391182 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.391184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.391754 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.391755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.391846 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.391849 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.391914 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.391960 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 23:06:17.392037 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.389355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55343 (local address 127.0.98.1:43175)
0924 23:06:17.389403 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.389404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.389415 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:17.389446 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.389447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.389456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.389539 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.389712 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.390330 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.391279 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.391651 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.391745 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.391756 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.391932 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.391971 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:06:17.392042 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.389307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49975 (local address 127.0.98.2:46549)
0924 23:06:17.389355 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.389358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.389575 (+ 217us) server_negotiation.cc:407] Connection header received
0924 23:06:17.389629 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.389631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.389642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.389711 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.389956 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.390410 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.391326 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.391681 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.391761 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.391770 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.391938 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.391973 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:17.392081 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.389231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46549 (local address 127.0.98.3:49975)
0924 23:06:17.389489 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.389499 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.389577 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.389713 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.389713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.389715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.389816 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.389818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.390506 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.390508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.391234 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.391235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.391762 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.391763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.391855 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.391859 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.391922 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.391964 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":57,"thread_start_us":158,"threads_started":1}
I20260924 23:06:17.392397 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6076bdde6bf4c0388a291fd14c282ba" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" is_pre_election: true
I20260924 23:06:17.392534 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6076bdde6bf4c0388a291fd14c282ba" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c9ee48c546407eb135fd383657fc63" is_pre_election: true
I20260924 23:06:17.392584 1655 raft_consensus.cc:2454] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 0.
I20260924 23:06:17.392726 1802 raft_consensus.cc:2454] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 0.
I20260924 23:06:17.392858 1874 leader_election.cc:302] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414c8cf789e1440082e81be44e33241a, b076e0eb2b6d4e8b85a1a04e6be213c4; no voters:
I20260924 23:06:17.393004 2018 raft_consensus.cc:2781] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:17.393069 2018 raft_consensus.cc:482] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:17.393105 2018 raft_consensus.cc:3037] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.394111 2018 raft_consensus.cc:504] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.394266 2018 leader_election.cc:288] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 election: Requested vote from peers b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175), e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549)
I20260924 23:06:17.394431 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6076bdde6bf4c0388a291fd14c282ba" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4"
I20260924 23:06:17.394421 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6076bdde6bf4c0388a291fd14c282ba" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c9ee48c546407eb135fd383657fc63"
I20260924 23:06:17.394505 1802 raft_consensus.cc:3037] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.394505 1655 raft_consensus.cc:3037] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.395375 1655 raft_consensus.cc:2454] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 1.
I20260924 23:06:17.395406 1802 raft_consensus.cc:2454] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 1.
I20260924 23:06:17.395558 1874 leader_election.cc:302] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414c8cf789e1440082e81be44e33241a, b076e0eb2b6d4e8b85a1a04e6be213c4; no voters:
I20260924 23:06:17.395684 2018 raft_consensus.cc:2781] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:17.395843 2018 raft_consensus.cc:686] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 1 LEADER]: Becoming Leader. State: Replica: 414c8cf789e1440082e81be44e33241a, State: Running, Role: LEADER
I20260924 23:06:17.395942 2018 consensus_queue.cc:237] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } }
I20260924 23:06:17.396843 1503 catalog_manager.cc:5495] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a reported cstate change: term changed from 0 to 1, leader changed from <none> to 414c8cf789e1440082e81be44e33241a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "414c8cf789e1440082e81be44e33241a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:17.406625 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.404356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.0.98.3:43035)
0924 23:06:17.404395 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.404397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.404425 (+ 28us) server_negotiation.cc:407] Connection header received
0924 23:06:17.404513 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.404514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.404530 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.404589 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:17.404708 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.405161 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.405933 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.406134 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.406187 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.406363 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.406451 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.406510 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.406550 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.406568 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:06:17.407961 1709 tablet.cc:2369] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:17.408378 1655 raft_consensus.cc:1261] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Refusing update from remote peer 414c8cf789e1440082e81be44e33241a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:17.408427 1802 raft_consensus.cc:1261] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Refusing update from remote peer 414c8cf789e1440082e81be44e33241a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:17.408725 2018 consensus_queue.cc:1035] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:17.408908 2022 consensus_queue.cc:1035] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:17.411299 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333032662662049792 to 7333032662614536192. Current Snapshot: MvccSnapshot[applied={T|T < 7333032662662049792}]
I20260924 23:06:17.411803 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333032662662049792 to 7333032662614536192. Current Snapshot: MvccSnapshot[applied={T|T < 7333032662662049792}]
I20260924 23:06:17.415119 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:17.415231 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:17.415990 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: Sending DeleteTablet for 3 replicas of tablet c6076bdde6bf4c0388a291fd14c282ba
I20260924 23:06:17.416329 1914 tablet_service.cc:1514] Processing DeleteTablet for tablet c6076bdde6bf4c0388a291fd14c282ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50608
I20260924 23:06:17.416374 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet c6076bdde6bf4c0388a291fd14c282ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35752
I20260924 23:06:17.416364 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet c6076bdde6bf4c0388a291fd14c282ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36860
I20260924 23:06:17.416589 2035 tablet_replica.cc:330] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: stopping tablet replica
I20260924 23:06:17.416672 2036 tablet_replica.cc:330] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: stopping tablet replica
I20260924 23:06:17.416754 2035 raft_consensus.cc:2229] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:17.416761 2036 raft_consensus.cc:2229] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:17.416873 2036 raft_consensus.cc:2258] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:17.416842 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:17.416992 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:17.417241 2035 raft_consensus.cc:2258] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:17.417464 2036 ts_tablet_manager.cc:1905] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:17.417824 2035 ts_tablet_manager.cc:1905] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:17.418223 1782 tablet_service.cc:1467] Processing CreateTablet for tablet b5e9dde725fb46f484be6c582fbcfa38 (DEFAULT_TABLE table=test-table [id=0049cd38532b453ba2f70220467e3f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:17.418342 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e9dde725fb46f484be6c582fbcfa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.419663 2013 tablet_bootstrap.cc:492] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Bootstrap starting.
I20260924 23:06:17.419729 1914 tablet_service.cc:1467] Processing CreateTablet for tablet b5e9dde725fb46f484be6c582fbcfa38 (DEFAULT_TABLE table=test-table [id=0049cd38532b453ba2f70220467e3f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:17.420089 2013 tablet_bootstrap.cc:654] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.420208 1642 tablet_service.cc:1467] Processing CreateTablet for tablet b5e9dde725fb46f484be6c582fbcfa38 (DEFAULT_TABLE table=test-table [id=0049cd38532b453ba2f70220467e3f50]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:17.420333 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e9dde725fb46f484be6c582fbcfa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.420518 1914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e9dde725fb46f484be6c582fbcfa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.420588 2036 ts_tablet_manager.cc:1918] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:17.420650 2036 log.cc:1196] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/c6076bdde6bf4c0388a291fd14c282ba
I20260924 23:06:17.420908 2036 ts_tablet_manager.cc:1939] T c6076bdde6bf4c0388a291fd14c282ba P e2c9ee48c546407eb135fd383657fc63: Deleting consensus metadata
I20260924 23:06:17.421295 1490 catalog_manager.cc:4841] TS e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549): tablet c6076bdde6bf4c0388a291fd14c282ba (table test-table [id=2f0dd8cf2cb64ddc923ccd5df102fa3e]) successfully deleted
I20260924 23:06:17.420516 2035 ts_tablet_manager.cc:1918] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:17.421442 2035 log.cc:1196] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/c6076bdde6bf4c0388a291fd14c282ba
I20260924 23:06:17.421586 2014 tablet_bootstrap.cc:492] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Bootstrap starting.
I20260924 23:06:17.421833 2012 tablet_bootstrap.cc:492] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Bootstrap starting.
I20260924 23:06:17.421983 2014 tablet_bootstrap.cc:654] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.422035 2037 tablet_replica.cc:330] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: stopping tablet replica
I20260924 23:06:17.422559 2037 raft_consensus.cc:2229] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:17.422675 2037 raft_consensus.cc:2258] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:17.422720 2012 tablet_bootstrap.cc:654] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.422892 2035 ts_tablet_manager.cc:1939] T c6076bdde6bf4c0388a291fd14c282ba P 414c8cf789e1440082e81be44e33241a: Deleting consensus metadata
I20260924 23:06:17.423134 2037 ts_tablet_manager.cc:1905] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:17.423184 2014 tablet_bootstrap.cc:492] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: No bootstrap required, opened a new log
I20260924 23:06:17.423228 2014 ts_tablet_manager.cc:1397] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:06:17.423290 1490 catalog_manager.cc:4841] TS 414c8cf789e1440082e81be44e33241a (127.0.98.3:43035): tablet c6076bdde6bf4c0388a291fd14c282ba (table test-table [id=2f0dd8cf2cb64ddc923ccd5df102fa3e]) successfully deleted
I20260924 23:06:17.423388 2014 raft_consensus.cc:348] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.423452 2014 raft_consensus.cc:374] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.423477 2014 raft_consensus.cc:729] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b076e0eb2b6d4e8b85a1a04e6be213c4, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.423533 2014 consensus_queue.cc:260] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.423651 2014 ts_tablet_manager.cc:1428] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.424191 2012 tablet_bootstrap.cc:492] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: No bootstrap required, opened a new log
I20260924 23:06:17.424242 2012 ts_tablet_manager.cc:1397] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:17.424396 2012 raft_consensus.cc:348] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.424463 2012 raft_consensus.cc:374] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.424476 2012 raft_consensus.cc:729] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414c8cf789e1440082e81be44e33241a, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.424536 2012 consensus_queue.cc:260] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.424619 2012 ts_tablet_manager.cc:1428] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.425089 2037 ts_tablet_manager.cc:1918] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:17.425159 2037 log.cc:1196] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/c6076bdde6bf4c0388a291fd14c282ba
I20260924 23:06:17.425415 2037 ts_tablet_manager.cc:1939] T c6076bdde6bf4c0388a291fd14c282ba P b076e0eb2b6d4e8b85a1a04e6be213c4: Deleting consensus metadata
I20260924 23:06:17.425819 1489 catalog_manager.cc:4841] TS b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175): tablet c6076bdde6bf4c0388a291fd14c282ba (table test-table [id=2f0dd8cf2cb64ddc923ccd5df102fa3e]) successfully deleted
I20260924 23:06:17.426251 2013 tablet_bootstrap.cc:492] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: No bootstrap required, opened a new log
I20260924 23:06:17.426294 2013 ts_tablet_manager.cc:1397] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260924 23:06:17.426466 2013 raft_consensus.cc:348] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.426524 2013 raft_consensus.cc:374] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.426538 2013 raft_consensus.cc:729] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c9ee48c546407eb135fd383657fc63, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.426594 2013 consensus_queue.cc:260] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.426684 2013 ts_tablet_manager.cc:1428] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:17.436782 2033 raft_consensus.cc:482] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:17.436860 2033 raft_consensus.cc:504] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.437034 2033 leader_election.cc:288] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175), e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549)
I20260924 23:06:17.437196 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e9dde725fb46f484be6c582fbcfa38" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c9ee48c546407eb135fd383657fc63" is_pre_election: true
I20260924 23:06:17.437215 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e9dde725fb46f484be6c582fbcfa38" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" is_pre_election: true
I20260924 23:06:17.437283 1802 raft_consensus.cc:2454] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 0.
I20260924 23:06:17.437283 1655 raft_consensus.cc:2454] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 0.
I20260924 23:06:17.437400 1874 leader_election.cc:302] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414c8cf789e1440082e81be44e33241a, b076e0eb2b6d4e8b85a1a04e6be213c4; no voters:
I20260924 23:06:17.437489 2033 raft_consensus.cc:2781] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:17.437525 2033 raft_consensus.cc:482] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:17.437543 2033 raft_consensus.cc:3037] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.438436 2033 raft_consensus.cc:504] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.438570 2033 leader_election.cc:288] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 election: Requested vote from peers b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175), e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549)
I20260924 23:06:17.438709 1655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e9dde725fb46f484be6c582fbcfa38" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4"
I20260924 23:06:17.438787 1655 raft_consensus.cc:3037] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.438766 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e9dde725fb46f484be6c582fbcfa38" candidate_uuid: "414c8cf789e1440082e81be44e33241a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c9ee48c546407eb135fd383657fc63"
I20260924 23:06:17.438843 1802 raft_consensus.cc:3037] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.439432 1655 raft_consensus.cc:2454] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 1.
I20260924 23:06:17.439591 1802 raft_consensus.cc:2454] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 414c8cf789e1440082e81be44e33241a in term 1.
I20260924 23:06:17.439752 1874 leader_election.cc:302] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414c8cf789e1440082e81be44e33241a, b076e0eb2b6d4e8b85a1a04e6be213c4; no voters:
I20260924 23:06:17.439852 2033 raft_consensus.cc:2781] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:17.439898 2033 raft_consensus.cc:686] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 1 LEADER]: Becoming Leader. State: Replica: 414c8cf789e1440082e81be44e33241a, State: Running, Role: LEADER
I20260924 23:06:17.439945 2033 consensus_queue.cc:237] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.440410 1503 catalog_manager.cc:5495] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a reported cstate change: term changed from 0 to 1, leader changed from <none> to 414c8cf789e1440082e81be44e33241a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "414c8cf789e1440082e81be44e33241a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } health_report { overall_health: HEALTHY } } }
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:17.478813 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.466760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.0.98.62:45683)
0924 23:06:17.466830 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.466833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.467016 (+ 183us) server_negotiation.cc:407] Connection header received
0924 23:06:17.467086 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.467086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.467096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.467143 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.467794 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.468306 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.469362 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.469539 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.469596 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.475960 (+ 6364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.475962 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.475962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.475971 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.477526 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.477984 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.477985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.477986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.478046 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.478239 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.478240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.478240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.478360 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.478437 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.478590 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.478700 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 23:06:17.478899 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.466739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45683 (local address 127.0.0.1:37448)
0924 23:06:17.466929 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.466942 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.467022 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.467158 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.467159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.467577 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:17.467680 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.467682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.468389 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.468390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.469259 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.469261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.469618 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.469619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.469675 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.474184 (+ 4509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:17.474198 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:17.475829 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:17.477613 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.477621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.477622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.477834 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.478113 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:17.478114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:17.478114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:17.478174 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:17.478428 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:17.478435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:17.478490 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.478569 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.478760 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":80,"thread_start_us":107,"threads_started":1}
I20260924 23:06:17.480914 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37448:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 23:06:17.481120 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:17.482661 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 03c8ab38cc724a96a60e92cba7c71a59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6a6482a4fbd41b7a49dff18709380c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:17.482745 1782 tablet_service.cc:1467] Processing CreateTablet for tablet 03c8ab38cc724a96a60e92cba7c71a59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6a6482a4fbd41b7a49dff18709380c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:17.482854 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c8ab38cc724a96a60e92cba7c71a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.482910 1782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c8ab38cc724a96a60e92cba7c71a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.484144 2014 tablet_bootstrap.cc:492] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: Bootstrap starting.
I20260924 23:06:17.484226 1914 tablet_service.cc:1467] Processing CreateTablet for tablet 03c8ab38cc724a96a60e92cba7c71a59 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d6a6482a4fbd41b7a49dff18709380c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:17.484359 1914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c8ab38cc724a96a60e92cba7c71a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:17.484583 2014 tablet_bootstrap.cc:654] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.485592 2012 tablet_bootstrap.cc:492] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: Bootstrap starting.
I20260924 23:06:17.485620 2013 tablet_bootstrap.cc:492] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: Bootstrap starting.
I20260924 23:06:17.485934 2012 tablet_bootstrap.cc:654] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.486090 2013 tablet_bootstrap.cc:654] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:17.486531 2012 tablet_bootstrap.cc:492] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: No bootstrap required, opened a new log
I20260924 23:06:17.486583 2012 ts_tablet_manager.cc:1397] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:17.486742 2012 raft_consensus.cc:348] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.486824 2012 raft_consensus.cc:374] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.486822 2014 tablet_bootstrap.cc:492] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: No bootstrap required, opened a new log
I20260924 23:06:17.486846 2012 raft_consensus.cc:729] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 414c8cf789e1440082e81be44e33241a, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.486873 2014 ts_tablet_manager.cc:1397] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 23:06:17.486892 2012 consensus_queue.cc:260] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.486956 2013 tablet_bootstrap.cc:492] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: No bootstrap required, opened a new log
I20260924 23:06:17.486996 2013 ts_tablet_manager.cc:1397] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:17.487139 2013 raft_consensus.cc:348] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.487214 2013 raft_consensus.cc:374] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.487236 2013 raft_consensus.cc:729] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c9ee48c546407eb135fd383657fc63, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.487284 2013 consensus_queue.cc:260] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.487370 2013 ts_tablet_manager.cc:1428] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.486989 2012 ts_tablet_manager.cc:1428] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:17.487464 2014 raft_consensus.cc:348] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.487530 2014 raft_consensus.cc:374] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:17.487550 2014 raft_consensus.cc:729] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b076e0eb2b6d4e8b85a1a04e6be213c4, State: Initialized, Role: FOLLOWER
I20260924 23:06:17.487545 2019 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.487600 2014 consensus_queue.cc:260] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.487640 2019 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.487691 2014 ts_tablet_manager.cc:1428] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:17.487771 2033 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.487757 2020 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.487843 2020 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.487838 2033 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.512910 2020 raft_consensus.cc:482] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:17.513028 2020 raft_consensus.cc:504] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.513392 2020 leader_election.cc:288] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549), 414c8cf789e1440082e81be44e33241a (127.0.98.3:43035)
W20260924 23:06:17.516882 1989 tablet.cc:2369] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:17.516992 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.513631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40437 (local address 127.0.98.2:46549)
0924 23:06:17.513679 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.513681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.513800 (+ 119us) server_negotiation.cc:407] Connection header received
0924 23:06:17.513988 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.513989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.514001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.514031 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.514260 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.515031 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.516014 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.516579 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.516653 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.516664 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.516873 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.516924 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:17.517563 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.513565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46549 (local address 127.0.98.1:40437)
0924 23:06:17.513729 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.513741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.513839 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.514097 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.514098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.514100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.514181 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.514183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.515116 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.515117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.515931 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.515934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.516669 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.516670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.516752 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.516758 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.517390 (+ 632us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.517436 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260924 23:06:17.518044 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c8ab38cc724a96a60e92cba7c71a59" candidate_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c9ee48c546407eb135fd383657fc63" is_pre_election: true
I20260924 23:06:17.518157 1802 raft_consensus.cc:2454] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b076e0eb2b6d4e8b85a1a04e6be213c4 in term 0.
I20260924 23:06:17.518164 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.513821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49649 (local address 127.0.98.3:43035)
0924 23:06:17.513909 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.513912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.513963 (+ 51us) server_negotiation.cc:407] Connection header received
0924 23:06:17.514034 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.514035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.514047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.514112 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:17.515145 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.515848 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.516767 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.517779 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.517851 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.517861 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.518072 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.518111 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:06:17.518371 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.513761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43035 (local address 127.0.98.1:49649)
0924 23:06:17.513909 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.513918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.513969 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.514114 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.514114 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.514115 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:17.514185 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.514186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.515955 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.515956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.516697 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.516699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.517866 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.517867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.517965 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.517970 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.518049 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.518086 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":16,"thread_start_us":43,"threads_started":1}
I20260924 23:06:17.518349 1594 leader_election.cc:302] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b076e0eb2b6d4e8b85a1a04e6be213c4, e2c9ee48c546407eb135fd383657fc63; no voters:
I20260924 23:06:17.518549 2020 raft_consensus.cc:2781] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:17.518585 2020 raft_consensus.cc:482] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:17.518612 2020 raft_consensus.cc:3037] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.518721 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c8ab38cc724a96a60e92cba7c71a59" candidate_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414c8cf789e1440082e81be44e33241a" is_pre_election: true
I20260924 23:06:17.518862 1942 raft_consensus.cc:2454] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b076e0eb2b6d4e8b85a1a04e6be213c4 in term 0.
I20260924 23:06:17.519461 2020 raft_consensus.cc:504] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.519603 2020 leader_election.cc:288] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [CANDIDATE]: Term 1 election: Requested vote from peers e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549), 414c8cf789e1440082e81be44e33241a (127.0.98.3:43035)
I20260924 23:06:17.519785 1942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c8ab38cc724a96a60e92cba7c71a59" candidate_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "414c8cf789e1440082e81be44e33241a"
I20260924 23:06:17.519869 1942 raft_consensus.cc:3037] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.520699 1942 raft_consensus.cc:2454] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b076e0eb2b6d4e8b85a1a04e6be213c4 in term 1.
I20260924 23:06:17.520807 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c8ab38cc724a96a60e92cba7c71a59" candidate_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c9ee48c546407eb135fd383657fc63"
I20260924 23:06:17.520879 1802 raft_consensus.cc:3037] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:17.520891 1594 leader_election.cc:302] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 414c8cf789e1440082e81be44e33241a, b076e0eb2b6d4e8b85a1a04e6be213c4; no voters:
I20260924 23:06:17.520978 2020 raft_consensus.cc:2781] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:17.521122 2020 raft_consensus.cc:686] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 LEADER]: Becoming Leader. State: Replica: b076e0eb2b6d4e8b85a1a04e6be213c4, State: Running, Role: LEADER
I20260924 23:06:17.521214 2020 consensus_queue.cc:237] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } }
I20260924 23:06:17.521584 1802 raft_consensus.cc:2454] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b076e0eb2b6d4e8b85a1a04e6be213c4 in term 1.
I20260924 23:06:17.521695 2053 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: TxnStatusTablet state changed. Reason: New leader b076e0eb2b6d4e8b85a1a04e6be213c4. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.521824 2053 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:17.521837 1503 catalog_manager.cc:5495] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:17.522127 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 23:06:17.522245 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 23:06:17.534972 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.531721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.0.98.1:43175)
0924 23:06:17.531762 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.531764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.531773 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:17.531862 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.531863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.531882 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.531957 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:17.532104 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.532860 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.534040 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.534353 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.534442 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.534622 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.534742 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.534816 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.534873 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.534894 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:17.534998 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.531659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43175 (local address 127.0.0.1:36114)
0924 23:06:17.531734 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.531746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.531786 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.531976 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.531977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.531978 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:17.532029 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.532032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.532950 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.532951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.533944 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.533947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.534445 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.534446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.534504 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.534520 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.534812 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.534819 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.534867 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.534916 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 23:06:17.536577 1802 raft_consensus.cc:1261] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Refusing update from remote peer b076e0eb2b6d4e8b85a1a04e6be213c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:17.536598 1942 raft_consensus.cc:1261] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Refusing update from remote peer b076e0eb2b6d4e8b85a1a04e6be213c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:17.536808 2020 consensus_queue.cc:1035] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:17.536976 2053 consensus_queue.cc:1035] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:17.537494 2033 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: TxnStatusTablet state changed. Reason: New leader b076e0eb2b6d4e8b85a1a04e6be213c4. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.537583 2033 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.537591 2019 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: TxnStatusTablet state changed. Reason: New leader b076e0eb2b6d4e8b85a1a04e6be213c4. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.537675 2019 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.538779 2020 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.538878 2020 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:17.538965 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333032663187845120 to 7333032663127412736. Current Snapshot: MvccSnapshot[applied={T|T < 7333032663187845120}]
I20260924 23:06:17.539079 2019 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.539145 2019 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P e2c9ee48c546407eb135fd383657fc63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.542455 2053 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.542538 2053 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P b076e0eb2b6d4e8b85a1a04e6be213c4: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:17.543426 2033 tablet_replica.cc:439] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 } } peers { permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 } } peers { permanent_uuid: "414c8cf789e1440082e81be44e33241a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43035 } } }
I20260924 23:06:17.543511 2033 tablet_replica.cc:442] T 03c8ab38cc724a96a60e92cba7c71a59 P 414c8cf789e1440082e81be44e33241a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:17.546875 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.543792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32935 (local address 127.0.98.1:43175)
0924 23:06:17.543832 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.543834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.543850 (+ 16us) server_negotiation.cc:407] Connection header received
0924 23:06:17.543872 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.543873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.543883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.543938 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:17.544123 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.544909 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.546089 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.546383 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.546463 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.546641 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.546674 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.546729 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.546792 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.546812 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:17.546922 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.543015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43175 (local address 127.0.98.3:32935)
0924 23:06:17.543143 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.543153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.543187 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.543981 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.543982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.543984 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:17.544027 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.544028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.544994 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.544995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.545979 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.545982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.546468 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.546469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.546538 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.546554 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.546741 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.546743 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.546790 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.546842 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 23:06:17.549155 1802 raft_consensus.cc:1261] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Refusing update from remote peer 414c8cf789e1440082e81be44e33241a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:17.549155 1655 raft_consensus.cc:1261] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Refusing update from remote peer 414c8cf789e1440082e81be44e33241a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:17.549348 2033 consensus_queue.cc:1035] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b076e0eb2b6d4e8b85a1a04e6be213c4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:17.549455 2033 consensus_queue.cc:1035] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c9ee48c546407eb135fd383657fc63" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:17.557843 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.555211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56999 (local address 127.0.98.3:43035)
0924 23:06:17.555272 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.555274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.555382 (+ 108us) server_negotiation.cc:407] Connection header received
0924 23:06:17.555454 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.555455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.555466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.555514 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:17.555717 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.556246 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.557144 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.557343 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.557422 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.557595 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.557628 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.557704 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.557758 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.557776 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:17.557909 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.555153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43035 (local address 127.0.98.1:56999)
0924 23:06:17.555307 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:17.555318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:17.555407 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:17.555551 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:17.555552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:17.555554 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:17.555616 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.555617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.556334 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.556336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.557053 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:17.557056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.557424 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.557425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:17.557486 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.557507 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:17.557697 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:17.557700 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:17.557753 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:06:17.557792 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260924 23:06:17.568392 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:17.568504 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:17.569114 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ab5529e0e3a46d8b9f478a01d8a0d7a: Sending DeleteTablet for 3 replicas of tablet b5e9dde725fb46f484be6c582fbcfa38
I20260924 23:06:17.569340 1782 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e9dde725fb46f484be6c582fbcfa38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35752
I20260924 23:06:17.569458 1914 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e9dde725fb46f484be6c582fbcfa38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50608
I20260924 23:06:17.569552 2036 tablet_replica.cc:330] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: stopping tablet replica
I20260924 23:06:17.569555 2035 tablet_replica.cc:330] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: stopping tablet replica
I20260924 23:06:17.569607 2036 raft_consensus.cc:2229] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:17.569653 2035 raft_consensus.cc:2229] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:17.569648 2036 raft_consensus.cc:2258] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:17.569720 2035 raft_consensus.cc:2258] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:17.569958 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet b5e9dde725fb46f484be6c582fbcfa38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36860
I20260924 23:06:17.570062 2036 ts_tablet_manager.cc:1905] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:17.571534 2037 tablet_replica.cc:330] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: stopping tablet replica
I20260924 23:06:17.571619 2037 raft_consensus.cc:2229] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:17.571668 2037 raft_consensus.cc:2258] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:17.572027 2036 ts_tablet_manager.cc:1918] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:17.572096 2036 log.cc:1196] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/b5e9dde725fb46f484be6c582fbcfa38
I20260924 23:06:17.572124 2035 ts_tablet_manager.cc:1905] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:17.572162 2037 ts_tablet_manager.cc:1905] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:17.572324 2036 ts_tablet_manager.cc:1939] T b5e9dde725fb46f484be6c582fbcfa38 P e2c9ee48c546407eb135fd383657fc63: Deleting consensus metadata
I20260924 23:06:17.572649 1490 catalog_manager.cc:4841] TS e2c9ee48c546407eb135fd383657fc63 (127.0.98.2:46549): tablet b5e9dde725fb46f484be6c582fbcfa38 (table test-table [id=0049cd38532b453ba2f70220467e3f50]) successfully deleted
I20260924 23:06:17.573746 2035 ts_tablet_manager.cc:1918] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:17.573763 2037 ts_tablet_manager.cc:1918] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:17.573808 2035 log.cc:1196] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/b5e9dde725fb46f484be6c582fbcfa38
I20260924 23:06:17.573815 2037 log.cc:1196] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/b5e9dde725fb46f484be6c582fbcfa38
I20260924 23:06:17.574051 2035 ts_tablet_manager.cc:1939] T b5e9dde725fb46f484be6c582fbcfa38 P 414c8cf789e1440082e81be44e33241a: Deleting consensus metadata
I20260924 23:06:17.574075 2037 ts_tablet_manager.cc:1939] T b5e9dde725fb46f484be6c582fbcfa38 P b076e0eb2b6d4e8b85a1a04e6be213c4: Deleting consensus metadata
I20260924 23:06:17.574380 1489 catalog_manager.cc:4841] TS b076e0eb2b6d4e8b85a1a04e6be213c4 (127.0.98.1:43175): tablet b5e9dde725fb46f484be6c582fbcfa38 (table test-table [id=0049cd38532b453ba2f70220467e3f50]) successfully deleted
I20260924 23:06:17.574460 1490 catalog_manager.cc:4841] TS 414c8cf789e1440082e81be44e33241a (127.0.98.3:43035): tablet b5e9dde725fb46f484be6c582fbcfa38 (table test-table [id=0049cd38532b453ba2f70220467e3f50]) successfully deleted
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291177, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:17.581138 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.570233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.0.98.1:43175)
0924 23:06:17.571745 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.571747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.571758 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:17.571779 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.571780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.571789 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.571848 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.572723 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.573411 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.574693 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.574897 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.574982 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.578290 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.578299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.578300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.578308 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.579910 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.580299 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.580303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.580303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.580366 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.580571 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.580572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.580572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.580729 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.580897 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.580936 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.581021 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1491}
W20260924 23:06:17.581439 1691 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36122
I20260924 23:06:17.588848 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:17.582474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.0.98.62:45683)
0924 23:06:17.582525 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:17.582527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:17.582539 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:17.582615 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:17.582616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:17.582626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:17.582683 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:17.583136 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.583943 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.584614 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:17.584916 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:17.584985 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:17.586255 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:17.586257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:17.586257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:17.586265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:17.587787 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.588113 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.588114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.588114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.588176 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:17.588368 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:17.588369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:17.588369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:17.588506 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:17.588603 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:17.588676 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:06:17.588751 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 23:06:17.589105 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37450
I20260924 23:06:17.589718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1576
I20260924 23:06:17.596242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1717
I20260924 23:06:17.601182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1857
I20260924 23:06:17.606866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 1472
2026-09-24T23:06:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2644 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[2074](info): set up 2 sockets
Sep 24 23:06:17 dist-test-slave-ph9j krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260924 23:06:18.642217 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:18 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:18.760617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:18.842739 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:18.843003 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:18.843039 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:18.844179 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:18.844259 2087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 23:06:18.844285 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:18.844317 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:18.844348 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:18.844375 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:18.845916 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34241
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:18.846211 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:18.846460 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:06:18.851692 2087 server_base.cc:1034] running on GCE node
W20260924 23:06:18.851779 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:18.851672 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:18.851702 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:18.852124 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:18.852456 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:18.853632 2087 hybrid_clock.cc:648] HybridClock initialized: now 1790291178853608 us; error 49 us; skew 500 ppm
Sep 24 23:06:18 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:18.856392 2087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:18.856844 2087 webserver.cc:460] Webserver started at http://127.0.98.62:41927/ using document root <none> and password file <none>
I20260924 23:06:18.857080 2087 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:18.857147 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:18.857303 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:18.858484 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "7a51d1c89c4d40ccb9c46ce132e03f05"
format_stamp: "Formatted at 2026-09-24 23:06:18 on dist-test-slave-ph9j"
server_key: "bff224577acf25fba49f2a8203aac24c"
server_key_iv: "9c6a122747a0aeaed42ee7e914dbbbe9"
server_key_version: "encryptionkey@0"
I20260924 23:06:18.858861 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a51d1c89c4d40ccb9c46ce132e03f05"
format_stamp: "Formatted at 2026-09-24 23:06:18 on dist-test-slave-ph9j"
server_key: "bff224577acf25fba49f2a8203aac24c"
server_key_iv: "9c6a122747a0aeaed42ee7e914dbbbe9"
server_key_version: "encryptionkey@0"
I20260924 23:06:18.860267 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:18.861042 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:18.861238 2087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:18.861346 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "7a51d1c89c4d40ccb9c46ce132e03f05"
format_stamp: "Formatted at 2026-09-24 23:06:18 on dist-test-slave-ph9j"
server_key: "bff224577acf25fba49f2a8203aac24c"
server_key_iv: "9c6a122747a0aeaed42ee7e914dbbbe9"
server_key_version: "encryptionkey@0"
I20260924 23:06:18.861420 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:18.884829 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:18.886948 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:18.887120 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:18.891799 2087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34241
I20260924 23:06:18.891855 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34241 every 8 connection(s)
I20260924 23:06:18.892208 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:18.892874 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:18.895124 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Bootstrap starting.
I20260924 23:06:18.895465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 2087
I20260924 23:06:18.895576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:18.895776 392 external_mini_cluster.cc:1428] Setting key 95d80e7d50e50fd18eb500a82980e866
I20260924 23:06:18.895823 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:18.896145 2165 log.cc:824] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:18.896903 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: No bootstrap required, opened a new log
I20260924 23:06:18.898662 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 23:06:18.898795 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:18.898818 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a51d1c89c4d40ccb9c46ce132e03f05, State: Initialized, Role: FOLLOWER
I20260924 23:06:18.898948 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 23:06:18.899029 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:18.899072 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:18.899118 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:18.899724 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 23:06:18.899828 2165 leader_election.cc:302] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a51d1c89c4d40ccb9c46ce132e03f05; no voters:
I20260924 23:06:18.899984 2165 leader_election.cc:288] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:18.900065 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:18.900234 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [term 1 LEADER]: Becoming Leader. State: Replica: 7a51d1c89c4d40ccb9c46ce132e03f05, State: Running, Role: LEADER
I20260924 23:06:18.900309 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:18.900321 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } }
I20260924 23:06:18.900655 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a51d1c89c4d40ccb9c46ce132e03f05. Latest consensus state: current_term: 1 leader_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 23:06:18.900717 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:18.900895 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a51d1c89c4d40ccb9c46ce132e03f05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34241 } } }
I20260924 23:06:18.900956 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:18.901341 2174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:18.901912 2174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 23:06:18 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:18.903914 2174 catalog_manager.cc:1347] Generated new cluster ID: 5941782a91e74405a95e929efac0649b
I20260924 23:06:18.903967 2174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:18.907284 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:18.896291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.0.98.62:34241)
0924 23:06:18.896454 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:18.896463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:18.896498 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:06:18.897033 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:18.897041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:18.897097 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:18.897224 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:18.897999 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:18.899431 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:18.900538 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:18.901170 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:18.901291 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:18.904100 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:18.904103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:18.904104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:18.904115 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:18.905785 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:18.906185 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:18.906188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:18.906189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:18.906280 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:18.906493 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:18.906493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:18.906494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:18.906718 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:18.906877 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:18.906955 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 23:06:18.907090 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260924 23:06:18.913182 2174 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:18.913671 2174 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:18.918442 2174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Generated new TSK 0
I20260924 23:06:18.918593 2174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:18.946398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:19.027956 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:19.028188 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:19.028235 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:19.029392 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:19.029459 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:19.029495 2191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 23:06:19.029525 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:19.029556 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:19.031087 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:19.031339 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:19.031563 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:19.036516 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:19.036605 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:19.036679 2191 server_base.cc:1034] running on GCE node
W20260924 23:06:19.036839 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:19.037077 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:19.037319 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:19.038478 2191 hybrid_clock.cc:648] HybridClock initialized: now 1790291179038456 us; error 56 us; skew 500 ppm
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:19.040823 2191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:19.041267 2191 webserver.cc:460] Webserver started at http://127.0.98.1:43039/ using document root <none> and password file <none>
I20260924 23:06:19.041484 2191 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:19.041563 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:19.041708 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:19.042812 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ad35880300f492cab7fd0f27e034013"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "be1aee9c62c4da342ae5df4580a064b8"
server_key_iv: "2a7c1d5bb091a407d5acc3866e8dfe78"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.043181 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ad35880300f492cab7fd0f27e034013"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "be1aee9c62c4da342ae5df4580a064b8"
server_key_iv: "2a7c1d5bb091a407d5acc3866e8dfe78"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.044464 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:19.045173 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.045346 2191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:19.045449 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "0ad35880300f492cab7fd0f27e034013"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "be1aee9c62c4da342ae5df4580a064b8"
server_key_iv: "2a7c1d5bb091a407d5acc3866e8dfe78"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.045533 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:19.071404 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:19.073410 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:19.073585 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:19.074191 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:19.074254 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.074290 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:19.074337 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.081521 2191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41845
I20260924 23:06:19.081597 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41845 every 8 connection(s)
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:19.081954 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:19.084726 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.074727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47495 (local address 127.0.98.62:34241)
0924 23:06:19.074806 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.074809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.075028 (+ 219us) server_negotiation.cc:407] Connection header received
0924 23:06:19.075556 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.075557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.075572 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.075656 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.076556 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.077401 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.079042 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.079448 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.079540 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.082442 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.082444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.082444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.082459 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.083640 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.084018 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.084019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.084019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.084072 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.084249 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.084250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.084250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.084352 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.084402 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.084591 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.084663 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:06:19.084970 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.074699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:47495)
0924 23:06:19.074890 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.074934 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.075446 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.075767 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.075771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.076122 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.076458 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.076464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.077511 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.077513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.078939 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.078942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.079576 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.079577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.079711 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.080319 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.080331 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.082336 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.083714 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.083719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.083721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.083944 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.084114 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.084114 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.084114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.084199 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.084392 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.084394 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.084505 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.084580 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.084699 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":119,"threads_started":1}
I20260924 23:06:19.085465 2322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 23:06:19.085551 2322 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:19.085742 2322 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 23:06:19.086294 2118 ts_manager.cc:194] Registered new tserver with Master: 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845)
I20260924 23:06:19.087287 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47495
I20260924 23:06:19.091400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 2191
I20260924 23:06:19.091493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:19.091630 392 external_mini_cluster.cc:1428] Setting key 9430c4b648eef01e00cff56faa8a4e92
I20260924 23:06:19.092267 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.086577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55303 (local address 127.0.98.62:34241)
0924 23:06:19.086619 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.086621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.086740 (+ 119us) server_negotiation.cc:407] Connection header received
0924 23:06:19.086808 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.086808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.086817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.086871 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.087202 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.087739 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.088621 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.088797 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.088844 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.089695 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.089696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.089697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.089703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.091126 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.091453 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.091455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.091455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.091516 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.091761 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.091762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.091763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.091908 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.091978 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.092144 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.092207 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:19.092329 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.086494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.1:55303)
0924 23:06:19.086680 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.086691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.086754 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.086880 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.086881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.087079 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.087138 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.087139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.087823 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.087824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.088548 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.088550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.088857 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.088857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.088906 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.089282 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.089286 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.089613 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.091203 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.091204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.091204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.091370 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.091620 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.091621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.091621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.091710 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.091993 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.091994 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.092042 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.092123 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.092227 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":27,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:19.116606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:19.196619 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:19.196828 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:19.196862 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:19.197971 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:19.198009 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:19.198028 2331 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 23:06:19.198040 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:19.198057 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:19.199427 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:19.199610 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:19.199785 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:19.205351 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:19.205300 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:19.205317 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:19.205466 2331 server_base.cc:1034] running on GCE node
I20260924 23:06:19.205786 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:19.206086 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:19.207289 2331 hybrid_clock.cc:648] HybridClock initialized: now 1790291179207300 us; error 52 us; skew 500 ppm
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:19.209995 2331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:19.210476 2331 webserver.cc:460] Webserver started at http://127.0.98.2:44447/ using document root <none> and password file <none>
I20260924 23:06:19.210717 2331 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:19.210789 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:19.210947 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:19.212008 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "9673028ac2c34773a3750f25e30f4a96"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "e9097759ec2efa342de0d9a066baae18"
server_key_iv: "0d34c35c8e29bf054888eed9d3146292"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.212381 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9673028ac2c34773a3750f25e30f4a96"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "e9097759ec2efa342de0d9a066baae18"
server_key_iv: "0d34c35c8e29bf054888eed9d3146292"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.213722 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:19.214600 2346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.214834 2331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:19.214933 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "9673028ac2c34773a3750f25e30f4a96"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "e9097759ec2efa342de0d9a066baae18"
server_key_iv: "0d34c35c8e29bf054888eed9d3146292"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.215015 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:19.268023 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:19.270499 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:19.270673 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:19.271229 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:19.271293 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.271341 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:19.271369 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.278523 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40331
I20260924 23:06:19.278777 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40331 every 8 connection(s)
I20260924 23:06:19.278936 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:19.281755 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 2331
I20260924 23:06:19.281857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:19.282051 392 external_mini_cluster.cc:1428] Setting key c3235d73c604d01e07caf38a4c908432
I20260924 23:06:19.282460 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.271812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44331 (local address 127.0.98.62:34241)
0924 23:06:19.271886 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.271888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.272093 (+ 205us) server_negotiation.cc:407] Connection header received
0924 23:06:19.272652 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.272653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.272665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.272810 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.273821 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.274418 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.276214 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.276548 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.276605 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.279570 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.279572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.279572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.279579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.280764 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.281223 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.281224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.281225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.281286 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.281579 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.281580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.281580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.281747 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.281867 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.282243 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.282351 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:06:19.282675 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.271802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:44331)
0924 23:06:19.271975 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.272015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.272558 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.272949 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.272956 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.273381 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.273697 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.273704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.274525 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.274526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.276111 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.276114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.276616 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.276617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.276794 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.277569 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.277581 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.279480 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.280857 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.280863 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.280865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.281093 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.281368 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.281369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.281369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.281476 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.281871 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.281877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.282063 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.282203 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.282415 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260924 23:06:19.284319 2462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 23:06:19.284405 2462 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:19.284605 2462 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 23:06:19.284974 2118 ts_manager.cc:194] Registered new tserver with Master: 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331)
I20260924 23:06:19.285470 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44331
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 23:06:19.291361 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.285486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38023 (local address 127.0.98.62:34241)
0924 23:06:19.285534 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.285536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.285701 (+ 165us) server_negotiation.cc:407] Connection header received
0924 23:06:19.285773 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.285774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.285794 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.285859 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.286193 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.286747 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.287467 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.287658 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.287740 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.288684 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.288686 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.288686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.288692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.290203 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.290547 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.290548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.290549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.290603 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.290824 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.290825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.290825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.291026 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.291090 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.291224 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.291299 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:06:19.291388 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.285408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.2:38023)
0924 23:06:19.285629 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.285639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.285715 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.285865 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.285866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.286065 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.286115 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.286117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.286841 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.286842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.287371 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.287373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.287742 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.287743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.287818 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.288280 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.288292 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.288580 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.290293 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.290294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.290295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.290478 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.290695 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.290696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.290696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.290753 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.291101 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.291102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.291147 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.291202 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.291313 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":173,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:19.309474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34241
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:19.391839 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:19.392050 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:19.392076 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:19.393168 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:19.393208 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:19.393229 2471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 23:06:19.393241 2471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:19.393258 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:19.394739 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:19.394924 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:19.395097 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:19.400311 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:19.400339 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:19.400427 2471 server_base.cc:1034] running on GCE node
W20260924 23:06:19.400327 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:19.400796 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:19.401113 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:19.402267 2471 hybrid_clock.cc:648] HybridClock initialized: now 1790291179402252 us; error 55 us; skew 500 ppm
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:19.404707 2471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:19.405135 2471 webserver.cc:460] Webserver started at http://127.0.98.3:41197/ using document root <none> and password file <none>
I20260924 23:06:19.405365 2471 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:19.405432 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:19.405578 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:19.406711 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c33b0b3e821435a835cbbec6b009098"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "46d99804fe0abe76ceeb5c9ea058c7b8"
server_key_iv: "3ee37f55d2b9d8909f892db7dba57586"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.407081 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c33b0b3e821435a835cbbec6b009098"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "46d99804fe0abe76ceeb5c9ea058c7b8"
server_key_iv: "3ee37f55d2b9d8909f892db7dba57586"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.408520 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:19.409255 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.409430 2471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:19.409531 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "6c33b0b3e821435a835cbbec6b009098"
format_stamp: "Formatted at 2026-09-24 23:06:19 on dist-test-slave-ph9j"
server_key: "46d99804fe0abe76ceeb5c9ea058c7b8"
server_key_iv: "3ee37f55d2b9d8909f892db7dba57586"
server_key_version: "encryptionkey@0"
I20260924 23:06:19.409601 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:19.430768 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:19.432765 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:19.432931 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:19.433530 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:19.433581 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.433614 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:19.433635 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.440865 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35549
I20260924 23:06:19.440924 2601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35549 every 8 connection(s)
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:19.441228 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:19.444249 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.433994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51439 (local address 127.0.98.62:34241)
0924 23:06:19.434048 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.434051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.434375 (+ 324us) server_negotiation.cc:407] Connection header received
0924 23:06:19.435114 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.435115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.435131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.435228 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.436136 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.436990 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.438698 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.439042 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.439128 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.441843 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.441845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.441845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.441851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.443007 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.443368 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.443369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.443370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.443418 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.443644 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.443645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.443646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.443812 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.443880 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.444119 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.444187 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:19.444401 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.434022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:51439)
0924 23:06:19.434215 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.434262 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.434982 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.435318 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.435323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.435731 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.436048 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.436054 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.437083 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.437084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.438583 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.438585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.439155 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.439156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.439265 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.439813 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.439824 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.441704 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.443076 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.443081 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.443083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.443301 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.443475 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.443475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.443476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.443566 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.443881 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.443885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.443992 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.444073 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.444194 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260924 23:06:19.444516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 2471
I20260924 23:06:19.444623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:19.444810 392 external_mini_cluster.cc:1428] Setting key 6cf3b22ed420945ce4c176b48a72ed92
I20260924 23:06:19.444964 2602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34241
I20260924 23:06:19.445070 2602 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:19.445261 2602 heartbeater.cc:511] Master 127.0.98.62:34241 requested a full tablet report, sending...
I20260924 23:06:19.445565 2118 ts_manager.cc:194] Registered new tserver with Master: 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.446103 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51439
I20260924 23:06:19.451344 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.445973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36093 (local address 127.0.98.62:34241)
0924 23:06:19.446011 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.446013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.446169 (+ 156us) server_negotiation.cc:407] Connection header received
0924 23:06:19.446288 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.446288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.446297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.446334 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.446666 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.447125 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.447914 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.448118 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.448200 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.449064 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.449065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.449065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.449072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.450484 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.450778 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.450779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.450780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.450827 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.450987 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.450987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.450988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.451074 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.451122 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.451237 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.451289 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:06:19.451362 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.445956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.98.3:36093)
0924 23:06:19.446107 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.446119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.446183 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.446359 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.446360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.446540 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.446586 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.446588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.447223 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.447224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.447834 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.447835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.448202 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.448203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.448281 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.448694 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.448706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.448959 (+ 253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.450559 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.450560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.450560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.450715 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.450883 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.450884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.450884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.450934 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.451126 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.451127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.451157 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.451207 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.451296 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":84,"threads_started":1}
I20260924 23:06:19.456459 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:19.479070 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.472001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.0.98.62:34241)
0924 23:06:19.472086 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.472087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.472094 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:19.472117 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.472118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.472126 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.472192 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.472779 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.473276 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.474076 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.474266 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.474357 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.476582 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.476584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.476585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.476593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.477954 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.478291 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.478293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.478294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.478352 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.478551 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.478552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.478553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.478665 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.478771 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.478888 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.478964 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 23:06:19.480656 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:19.482270 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:19.489214 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.484918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.0.98.1:41845)
0924 23:06:19.485094 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.485097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.485290 (+ 193us) server_negotiation.cc:407] Connection header received
0924 23:06:19.485333 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.485335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.485351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.485432 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.485637 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.486832 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488037 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.488771 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488842 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.488897 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.489066 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.489104 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260924 23:06:19.489238 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.484593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40331 (local address 127.0.0.1:39108)
0924 23:06:19.484715 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.484758 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.484827 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.485177 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.485180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.485183 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.485277 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.485279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.486721 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.486722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.488132 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.488134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.488967 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.489020 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.489023 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.489066 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.489102 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":82,"thread_start_us":72,"threads_started":1}
I20260924 23:06:19.489259 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.484595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41845 (local address 127.0.0.1:55694)
0924 23:06:19.485104 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.485112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.485185 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.485499 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.485500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.485502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.485559 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.485561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.486917 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.486918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.487940 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.487943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.488932 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.488995 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.488999 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.489051 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.489094 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":112,"threads_started":1}
I20260924 23:06:19.489311 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.484791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39108 (local address 127.0.98.2:40331)
0924 23:06:19.484987 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.484992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.485003 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:19.485046 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.485049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.485066 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.485153 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.485367 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.486632 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488217 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.488893 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488953 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.489000 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.489163 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.489208 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260924 23:06:19.489751 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.484852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.98.3:35549)
0924 23:06:19.485009 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.485014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.485370 (+ 356us) server_negotiation.cc:407] Connection header received
0924 23:06:19.485493 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.485495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.485510 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.485579 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.485750 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.486655 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488231 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.488710 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488811 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.488875 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.489616 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.489657 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260924 23:06:19.489118 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.484600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35549 (local address 127.0.0.1:55922)
0924 23:06:19.485248 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.485258 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.485308 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.485637 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.485638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.485640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.485667 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.485668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.486766 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.486767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.488132 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.488134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.488814 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.488815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.488898 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.488912 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.488980 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.489037 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":40,"thread_start_us":83,"threads_started":1}
I20260924 23:06:19.490082 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 5e61adf66320436d9041dea057407c36 (DEFAULT_TABLE table=test-table [id=5054926d84cf45ed905cf39960fecc4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:19.490518 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61adf66320436d9041dea057407c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.490515 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 5e61adf66320436d9041dea057407c36 (DEFAULT_TABLE table=test-table [id=5054926d84cf45ed905cf39960fecc4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:19.490836 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61adf66320436d9041dea057407c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.494410 2626 tablet_bootstrap.cc:492] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Bootstrap starting.
I20260924 23:06:19.494575 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 5e61adf66320436d9041dea057407c36 (DEFAULT_TABLE table=test-table [id=5054926d84cf45ed905cf39960fecc4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:19.494949 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61adf66320436d9041dea057407c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.495060 2626 tablet_bootstrap.cc:654] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.495420 2626 log.cc:824] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:19.495746 2625 tablet_bootstrap.cc:492] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Bootstrap starting.
I20260924 23:06:19.496178 2626 tablet_bootstrap.cc:492] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: No bootstrap required, opened a new log
I20260924 23:06:19.496260 2626 ts_tablet_manager.cc:1397] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:19.496366 2625 tablet_bootstrap.cc:654] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.496680 2625 log.cc:824] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:19.497418 2625 tablet_bootstrap.cc:492] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: No bootstrap required, opened a new log
I20260924 23:06:19.497488 2625 ts_tablet_manager.cc:1397] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:19.498020 2626 raft_consensus.cc:348] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.498190 2626 raft_consensus.cc:374] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.498236 2626 raft_consensus.cc:729] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9673028ac2c34773a3750f25e30f4a96, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.498364 2626 consensus_queue.cc:260] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.498661 2626 ts_tablet_manager.cc:1428] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:06:19.498694 2462 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 23:06:19.499539 2625 raft_consensus.cc:348] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.499718 2625 raft_consensus.cc:374] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.499769 2625 raft_consensus.cc:729] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.499904 2625 consensus_queue.cc:260] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.500135 2625 ts_tablet_manager.cc:1428] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:19.500201 2322 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 23:06:19.500347 2629 tablet_bootstrap.cc:492] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Bootstrap starting.
I20260924 23:06:19.501312 2629 tablet_bootstrap.cc:654] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.501616 2629 log.cc:824] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:19.502429 2629 tablet_bootstrap.cc:492] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: No bootstrap required, opened a new log
I20260924 23:06:19.502490 2629 ts_tablet_manager.cc:1397] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:19.503825 2629 raft_consensus.cc:348] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.503939 2629 raft_consensus.cc:374] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.503959 2629 raft_consensus.cc:729] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c33b0b3e821435a835cbbec6b009098, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.504093 2629 consensus_queue.cc:260] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.504356 2629 ts_tablet_manager.cc:1428] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:19.504417 2602 heartbeater.cc:503] Master 127.0.98.62:34241 was elected leader, sending a full tablet report...
I20260924 23:06:19.520187 2631 raft_consensus.cc:482] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:19.520294 2631 raft_consensus.cc:504] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.520711 2631 leader_election.cc:288] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.523613 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.520977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59911 (local address 127.0.98.2:40331)
0924 23:06:19.521037 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.521039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.521049 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:19.521076 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.521077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.521088 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.521183 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.521345 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.521996 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.522906 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.523270 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.523362 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.523373 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.523522 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.523558 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:06:19.523612 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.520881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40331 (local address 127.0.98.1:59911)
0924 23:06:19.520939 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.520952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.520999 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.521191 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.521192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.521194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.521267 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.521268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.522084 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.522085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.522815 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.522817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.523363 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.523364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.523439 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.523443 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.523504 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.523541 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:06:19.523697 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.521049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36665 (local address 127.0.98.3:35549)
0924 23:06:19.521091 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.521092 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.521137 (+ 45us) server_negotiation.cc:407] Connection header received
0924 23:06:19.521195 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.521196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.521204 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.521266 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.521419 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.522061 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.522953 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.523315 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.523414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.523424 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.523584 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.523618 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:19.523916 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.520921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35549 (local address 127.0.98.1:36665)
0924 23:06:19.521094 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.521102 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.521146 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.521270 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.521271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.521272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.521348 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.521350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.522138 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.522139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.522850 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.522852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.523427 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.523429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.523502 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.523505 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.523576 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.523614 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260924 23:06:19.523916 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61adf66320436d9041dea057407c36" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96" is_pre_election: true
I20260924 23:06:19.524045 2416 raft_consensus.cc:2454] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:19.524191 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61adf66320436d9041dea057407c36" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098" is_pre_election: true
I20260924 23:06:19.524241 2210 leader_election.cc:302] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:19.524338 2556 raft_consensus.cc:2454] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:19.524364 2631 raft_consensus.cc:2781] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:19.524444 2631 raft_consensus.cc:482] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:19.524477 2631 raft_consensus.cc:3037] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.525162 2631 raft_consensus.cc:504] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.525295 2631 leader_election.cc:288] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Requested vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.525434 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61adf66320436d9041dea057407c36" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098"
I20260924 23:06:19.525434 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61adf66320436d9041dea057407c36" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96"
I20260924 23:06:19.525509 2556 raft_consensus.cc:3037] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.525511 2416 raft_consensus.cc:3037] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.526394 2416 raft_consensus.cc:2454] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:19.526394 2556 raft_consensus.cc:2454] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:19.526553 2210 leader_election.cc:302] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:19.526690 2631 raft_consensus.cc:2781] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:19.526831 2631 raft_consensus.cc:686] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Running, Role: LEADER
I20260924 23:06:19.526933 2631 consensus_queue.cc:237] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.527711 2118 catalog_manager.cc:5495] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ad35880300f492cab7fd0f27e034013 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ad35880300f492cab7fd0f27e034013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } health_report { overall_health: UNKNOWN } } }
W20260924 23:06:19.529444 2463 tablet.cc:2369] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 23:06:19 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:19.561188 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.554634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.0.98.62:34241)
0924 23:06:19.554708 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.554709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.554787 (+ 78us) server_negotiation.cc:407] Connection header received
0924 23:06:19.554812 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.554813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.554821 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.554880 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:19.555308 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.555794 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.556458 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.556628 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.556675 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.558760 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:19.558761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:19.558761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:19.558767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:19.560210 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.560548 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.560549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.560549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.560597 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:19.560761 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:19.560762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:19.560762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:19.560900 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:19.560951 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.561064 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.561127 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 23:06:19.561327 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.554544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34241 (local address 127.0.0.1:57798)
0924 23:06:19.554705 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.554714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.554770 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.554901 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.554902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.555195 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:19.555240 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.555241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.555873 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.555874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.556388 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.556389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.556689 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.556690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.556726 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.557215 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:19.557219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:19.558673 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:19.560285 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.560290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.560292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.560477 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.560656 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:19.560657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:19.560657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:19.560710 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:19.560955 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:19.560960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:19.560999 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.561050 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.561177 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":44,"thread_start_us":84,"threads_started":1}
I20260924 23:06:19.563268 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57798:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 23:06:19.563468 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:19.564914 2256 tablet_service.cc:1467] Processing CreateTablet for tablet df6c08dd0732435baa1267c276d77b7a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ac80af46923471a8027dbb51c05e310]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:19.564949 2536 tablet_service.cc:1467] Processing CreateTablet for tablet df6c08dd0732435baa1267c276d77b7a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ac80af46923471a8027dbb51c05e310]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:19.564992 2396 tablet_service.cc:1467] Processing CreateTablet for tablet df6c08dd0732435baa1267c276d77b7a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ac80af46923471a8027dbb51c05e310]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:19.565100 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6c08dd0732435baa1267c276d77b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.565112 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6c08dd0732435baa1267c276d77b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.565112 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df6c08dd0732435baa1267c276d77b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.566408 2625 tablet_bootstrap.cc:492] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: Bootstrap starting.
I20260924 23:06:19.566636 2629 tablet_bootstrap.cc:492] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: Bootstrap starting.
I20260924 23:06:19.566879 2626 tablet_bootstrap.cc:492] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: Bootstrap starting.
I20260924 23:06:19.567267 2625 tablet_bootstrap.cc:654] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.567273 2629 tablet_bootstrap.cc:654] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.567451 2626 tablet_bootstrap.cc:654] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.567992 2629 tablet_bootstrap.cc:492] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: No bootstrap required, opened a new log
I20260924 23:06:19.568044 2629 ts_tablet_manager.cc:1397] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:19.568120 2625 tablet_bootstrap.cc:492] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: No bootstrap required, opened a new log
I20260924 23:06:19.568172 2626 tablet_bootstrap.cc:492] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: No bootstrap required, opened a new log
I20260924 23:06:19.568169 2625 ts_tablet_manager.cc:1397] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:19.568221 2626 ts_tablet_manager.cc:1397] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:19.568195 2629 raft_consensus.cc:348] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.568276 2629 raft_consensus.cc:374] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.568305 2629 raft_consensus.cc:729] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c33b0b3e821435a835cbbec6b009098, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.568356 2629 consensus_queue.cc:260] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.568333 2625 raft_consensus.cc:348] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.568379 2626 raft_consensus.cc:348] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.568432 2625 raft_consensus.cc:374] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.568442 2626 raft_consensus.cc:374] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.568454 2629 ts_tablet_manager.cc:1428] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:19.568465 2625 raft_consensus.cc:729] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.568466 2626 raft_consensus.cc:729] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9673028ac2c34773a3750f25e30f4a96, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.568508 2626 consensus_queue.cc:260] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.568508 2625 consensus_queue.cc:260] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.568594 2626 ts_tablet_manager.cc:1428] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:19.568596 2625 ts_tablet_manager.cc:1428] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:19.568739 2630 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.568751 2633 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.568840 2633 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:19.568840 2630 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:19.568961 2631 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.569046 2631 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 23:06:19.582715 2323 tablet.cc:2369] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:19.609006 2630 raft_consensus.cc:482] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:19.609099 2630 raft_consensus.cc:504] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.609463 2630 leader_election.cc:288] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.612787 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.609709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55481 (local address 127.0.98.1:41845)
0924 23:06:19.609757 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.609760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.609813 (+ 53us) server_negotiation.cc:407] Connection header received
0924 23:06:19.609899 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.609900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.609914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.609964 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.610120 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.610618 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.611938 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.612353 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.612448 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.612457 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.612631 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.612670 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:19.612787 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.609634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41845 (local address 127.0.98.2:55481)
0924 23:06:19.609742 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.609754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.609811 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.609966 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.609967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.609968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.610046 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.610048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.610772 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.610773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.611830 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.611833 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.612465 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.612466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.612556 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.612561 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.612617 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.612661 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":51}
I20260924 23:06:19.612908 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.609768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38585 (local address 127.0.98.3:35549)
0924 23:06:19.609820 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.609822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.609952 (+ 130us) server_negotiation.cc:407] Connection header received
0924 23:06:19.610038 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.610039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.610052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.610114 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:19.610277 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.610764 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.612095 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.612488 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.612575 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.612587 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.612772 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.612821 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:06:19.613039 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.609680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35549 (local address 127.0.98.2:38585)
0924 23:06:19.609857 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.609871 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.609968 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.610129 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.610130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.610132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:19.610199 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.610200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.610859 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.610860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.611994 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.611996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.612617 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.612618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.612703 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.612707 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.612765 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.612802 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260924 23:06:19.613231 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6c08dd0732435baa1267c276d77b7a" candidate_uuid: "9673028ac2c34773a3750f25e30f4a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad35880300f492cab7fd0f27e034013" is_pre_election: true
I20260924 23:06:19.613365 2276 raft_consensus.cc:2454] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9673028ac2c34773a3750f25e30f4a96 in term 0.
I20260924 23:06:19.613329 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6c08dd0732435baa1267c276d77b7a" candidate_uuid: "9673028ac2c34773a3750f25e30f4a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098" is_pre_election: true
I20260924 23:06:19.613415 2556 raft_consensus.cc:2454] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9673028ac2c34773a3750f25e30f4a96 in term 0.
I20260924 23:06:19.613556 2349 leader_election.cc:302] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:19.613705 2630 raft_consensus.cc:2781] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:19.613750 2630 raft_consensus.cc:482] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:19.613770 2630 raft_consensus.cc:3037] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.614490 2630 raft_consensus.cc:504] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.614629 2630 leader_election.cc:288] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [CANDIDATE]: Term 1 election: Requested vote from peers 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.614754 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6c08dd0732435baa1267c276d77b7a" candidate_uuid: "9673028ac2c34773a3750f25e30f4a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad35880300f492cab7fd0f27e034013"
I20260924 23:06:19.614789 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df6c08dd0732435baa1267c276d77b7a" candidate_uuid: "9673028ac2c34773a3750f25e30f4a96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098"
I20260924 23:06:19.614850 2276 raft_consensus.cc:3037] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.614862 2556 raft_consensus.cc:3037] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.615456 2276 raft_consensus.cc:2454] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9673028ac2c34773a3750f25e30f4a96 in term 1.
I20260924 23:06:19.615460 2556 raft_consensus.cc:2454] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9673028ac2c34773a3750f25e30f4a96 in term 1.
I20260924 23:06:19.615628 2348 leader_election.cc:302] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c33b0b3e821435a835cbbec6b009098, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:19.615742 2630 raft_consensus.cc:2781] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:19.615890 2630 raft_consensus.cc:686] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [term 1 LEADER]: Becoming Leader. State: Replica: 9673028ac2c34773a3750f25e30f4a96, State: Running, Role: LEADER
I20260924 23:06:19.615989 2630 consensus_queue.cc:237] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.616415 2647 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: TxnStatusTablet state changed. Reason: New leader 9673028ac2c34773a3750f25e30f4a96. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.616535 2647 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:19.616787 2117 catalog_manager.cc:5495] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:19.617103 2649 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 23:06:19.617225 2649 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 23:06:19.628523 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.625856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.0.98.2:40331)
0924 23:06:19.625921 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.625923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.625936 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:06:19.626056 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.626057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.626074 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.626140 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:19.626283 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.626830 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.627595 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.627891 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.627973 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.628167 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.628275 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.628368 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.628409 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.628429 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:06:19.628525 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.625785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40331 (local address 127.0.0.1:39114)
0924 23:06:19.625864 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.625902 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.625957 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.626144 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.626145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.626146 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:19.626202 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.626204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.626913 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.626914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.627506 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.627508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.627972 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.627973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.628039 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.628061 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.628344 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.628353 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.628403 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.628453 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:06:19.630537 2556 raft_consensus.cc:1261] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Refusing update from remote peer 9673028ac2c34773a3750f25e30f4a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:19.630524 2276 raft_consensus.cc:1261] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Refusing update from remote peer 9673028ac2c34773a3750f25e30f4a96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:19.630805 2647 consensus_queue.cc:1035] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:19.630942 2630 consensus_queue.cc:1035] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:19.632050 2631 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: TxnStatusTablet state changed. Reason: New leader 9673028ac2c34773a3750f25e30f4a96. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.632148 2631 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:19.632143 2633 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: TxnStatusTablet state changed. Reason: New leader 9673028ac2c34773a3750f25e30f4a96. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.632239 2633 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:19.633298 2630 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.633399 2630 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:19.633437 2633 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.633512 2633 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 6c33b0b3e821435a835cbbec6b009098: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:19.634320 2630 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.634410 2630 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 9673028ac2c34773a3750f25e30f4a96: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:19.634742 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333032671764086784 to 7333032671707652096. Current Snapshot: MvccSnapshot[applied={T|T < 7333032671764086784}]
I20260924 23:06:19.634902 2631 tablet_replica.cc:439] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9673028ac2c34773a3750f25e30f4a96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } } }
I20260924 23:06:19.634992 2631 tablet_replica.cc:442] T df6c08dd0732435baa1267c276d77b7a P 0ad35880300f492cab7fd0f27e034013: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:19.635128 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333032671764086784 to 7333032671707652096. Current Snapshot: MvccSnapshot[applied={T|T < 7333032671764086784}]
I20260924 23:06:19.638439 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.635968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55702 (local address 127.0.98.1:41845)
0924 23:06:19.636004 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.636006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.636062 (+ 56us) server_negotiation.cc:407] Connection header received
0924 23:06:19.636149 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.636150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.636167 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.636265 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:19.636393 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.636856 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.637628 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.637837 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.637946 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.638119 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.638228 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.638319 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.638341 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.638364 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 23:06:19.643635 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.640952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40669 (local address 127.0.98.2:40331)
0924 23:06:19.640996 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.640998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.641066 (+ 68us) server_negotiation.cc:407] Connection header received
0924 23:06:19.641136 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.641137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.641155 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.641203 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:19.641326 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.641804 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.642945 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.643129 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.643189 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.643385 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.643415 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.643457 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.643530 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.643548 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:19.643662 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.640855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40331 (local address 127.0.98.1:40669)
0924 23:06:19.641007 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.641018 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.641078 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.641207 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.641208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.641209 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:19.641266 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.641268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.641900 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.641901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.642845 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.642848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.643206 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.643207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.643273 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.643287 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.643478 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.643480 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.643528 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.643572 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260924 23:06:19.645938 2556 raft_consensus.cc:1261] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:19.646060 2416 raft_consensus.cc:1261] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:19.646209 2635 consensus_queue.cc:1035] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:19.646489 2635 consensus_queue.cc:1035] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:19.648115 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333032671828209664 to 7333032671342882816. Current Snapshot: MvccSnapshot[applied={T|T < 7333032671828209664}]
I20260924 23:06:19.655032 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.652455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40511 (local address 127.0.98.1:41845)
0924 23:06:19.652492 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:19.652494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:19.652600 (+ 106us) server_negotiation.cc:407] Connection header received
0924 23:06:19.652620 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:19.652621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:19.652631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:19.652694 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:19.652824 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.653270 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.654361 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.654630 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.654694 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.654852 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.654878 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.654919 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:19.654971 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 23:06:19.654985 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:06:19.655087 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:19.652431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41845 (local address 127.0.98.2:40511)
0924 23:06:19.652499 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:19.652509 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:19.652550 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:19.652701 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:19.652702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:19.652703 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:19.652766 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.652768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.653339 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.653340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.654280 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:19.654283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:19.654692 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:19.654692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:19.654756 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:19.654771 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:19.654920 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:19.654921 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:06:19.654969 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:06:19.655008 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:06:19.665141 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:19.665254 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:19.665974 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Sending DeleteTablet for 3 replicas of tablet 5e61adf66320436d9041dea057407c36
I20260924 23:06:19.666260 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e61adf66320436d9041dea057407c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39108
I20260924 23:06:19.666278 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e61adf66320436d9041dea057407c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55922
I20260924 23:06:19.666478 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e61adf66320436d9041dea057407c36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55694
I20260924 23:06:19.666499 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:19.666639 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:19.667009 2673 tablet_replica.cc:330] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: stopping tablet replica
I20260924 23:06:19.667114 2673 raft_consensus.cc:2229] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:19.667163 2673 raft_consensus.cc:2258] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:19.667865 2396 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd9d7b20974985a8eaf83d8969810d (DEFAULT_TABLE table=test-table [id=63d48c59db3547a8bd649673bfcbb4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:19.667927 2536 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd9d7b20974985a8eaf83d8969810d (DEFAULT_TABLE table=test-table [id=63d48c59db3547a8bd649673bfcbb4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:19.668000 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cd9d7b20974985a8eaf83d8969810d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.668041 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cd9d7b20974985a8eaf83d8969810d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.668294 2673 ts_tablet_manager.cc:1905] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:19.669478 2256 tablet_service.cc:1467] Processing CreateTablet for tablet f7cd9d7b20974985a8eaf83d8969810d (DEFAULT_TABLE table=test-table [id=63d48c59db3547a8bd649673bfcbb4f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:19.669595 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7cd9d7b20974985a8eaf83d8969810d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:19.670148 2673 ts_tablet_manager.cc:1918] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:19.670233 2673 log.cc:1196] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/5e61adf66320436d9041dea057407c36
I20260924 23:06:19.670522 2673 ts_tablet_manager.cc:1939] T 5e61adf66320436d9041dea057407c36 P 9673028ac2c34773a3750f25e30f4a96: Deleting consensus metadata
I20260924 23:06:19.671044 2625 tablet_bootstrap.cc:492] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Bootstrap starting.
I20260924 23:06:19.671072 2629 tablet_bootstrap.cc:492] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Bootstrap starting.
I20260924 23:06:19.671449 2625 tablet_bootstrap.cc:654] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.671473 2629 tablet_bootstrap.cc:654] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.671809 2626 tablet_bootstrap.cc:492] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Bootstrap starting.
I20260924 23:06:19.672119 2629 tablet_bootstrap.cc:492] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: No bootstrap required, opened a new log
I20260924 23:06:19.672165 2629 ts_tablet_manager.cc:1397] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:19.672180 2626 tablet_bootstrap.cc:654] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:19.672298 2629 raft_consensus.cc:348] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.672376 2629 raft_consensus.cc:374] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.672397 2629 raft_consensus.cc:729] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c33b0b3e821435a835cbbec6b009098, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.672438 2629 consensus_queue.cc:260] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.672519 2629 ts_tablet_manager.cc:1428] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.672711 2626 tablet_bootstrap.cc:492] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: No bootstrap required, opened a new log
I20260924 23:06:19.672708 2106 catalog_manager.cc:4841] TS 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331): tablet 5e61adf66320436d9041dea057407c36 (table test-table [id=5054926d84cf45ed905cf39960fecc4f]) successfully deleted
I20260924 23:06:19.672760 2626 ts_tablet_manager.cc:1397] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:19.672905 2626 raft_consensus.cc:348] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.672973 2626 raft_consensus.cc:374] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.672993 2626 raft_consensus.cc:729] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9673028ac2c34773a3750f25e30f4a96, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.673030 2626 consensus_queue.cc:260] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.673105 2626 ts_tablet_manager.cc:1428] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:19.674001 2672 tablet_replica.cc:330] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: stopping tablet replica
I20260924 23:06:19.674157 2672 raft_consensus.cc:2229] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:19.674274 2672 raft_consensus.cc:2258] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:19.674757 2625 tablet_bootstrap.cc:492] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: No bootstrap required, opened a new log
I20260924 23:06:19.674803 2625 ts_tablet_manager.cc:1397] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 23:06:19.674803 2672 ts_tablet_manager.cc:1905] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:19.674940 2625 raft_consensus.cc:348] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.675000 2625 raft_consensus.cc:374] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:19.675020 2625 raft_consensus.cc:729] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Initialized, Role: FOLLOWER
I20260924 23:06:19.675076 2625 consensus_queue.cc:260] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.675153 2625 ts_tablet_manager.cc:1428] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:19.676896 2672 ts_tablet_manager.cc:1918] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:19.676977 2672 log.cc:1196] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/5e61adf66320436d9041dea057407c36
I20260924 23:06:19.677246 2672 ts_tablet_manager.cc:1939] T 5e61adf66320436d9041dea057407c36 P 6c33b0b3e821435a835cbbec6b009098: Deleting consensus metadata
I20260924 23:06:19.677670 2104 catalog_manager.cc:4841] TS 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549): tablet 5e61adf66320436d9041dea057407c36 (table test-table [id=5054926d84cf45ed905cf39960fecc4f]) successfully deleted
I20260924 23:06:19.683772 2674 tablet_replica.cc:330] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: stopping tablet replica
I20260924 23:06:19.683916 2674 raft_consensus.cc:2229] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:19.684028 2674 raft_consensus.cc:2258] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:19.684577 2674 ts_tablet_manager.cc:1905] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:19.688555 2674 ts_tablet_manager.cc:1918] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:19.688642 2674 log.cc:1196] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/5e61adf66320436d9041dea057407c36
I20260924 23:06:19.688902 2674 ts_tablet_manager.cc:1939] T 5e61adf66320436d9041dea057407c36 P 0ad35880300f492cab7fd0f27e034013: Deleting consensus metadata
I20260924 23:06:19.689366 2105 catalog_manager.cc:4841] TS 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845): tablet 5e61adf66320436d9041dea057407c36 (table test-table [id=5054926d84cf45ed905cf39960fecc4f]) successfully deleted
W20260924 23:06:19.691874 2603 tablet.cc:2369] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:19.746157 2668 raft_consensus.cc:482] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:19.746259 2668 raft_consensus.cc:504] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.746775 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd9d7b20974985a8eaf83d8969810d" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96" is_pre_election: true
I20260924 23:06:19.746891 2416 raft_consensus.cc:2454] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:19.746991 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd9d7b20974985a8eaf83d8969810d" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098" is_pre_election: true
I20260924 23:06:19.747081 2556 raft_consensus.cc:2454] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:19.747298 2210 leader_election.cc:302] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:19.747396 2635 raft_consensus.cc:2781] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:19.747465 2635 raft_consensus.cc:482] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:19.747493 2635 raft_consensus.cc:3037] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.748488 2635 raft_consensus.cc:504] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.748624 2635 leader_election.cc:288] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Requested vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.748809 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd9d7b20974985a8eaf83d8969810d" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96"
I20260924 23:06:19.748843 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7cd9d7b20974985a8eaf83d8969810d" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098"
I20260924 23:06:19.748893 2416 raft_consensus.cc:3037] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.748915 2556 raft_consensus.cc:3037] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:19.749645 2556 raft_consensus.cc:2454] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:19.749740 2416 raft_consensus.cc:2454] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:19.749818 2208 leader_election.cc:302] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 6c33b0b3e821435a835cbbec6b009098; no voters:
I20260924 23:06:19.749917 2635 raft_consensus.cc:2781] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:19.749955 2635 raft_consensus.cc:686] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Running, Role: LEADER
I20260924 23:06:19.749995 2635 consensus_queue.cc:237] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:19.750586 2118 catalog_manager.cc:5495] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ad35880300f492cab7fd0f27e034013 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ad35880300f492cab7fd0f27e034013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:19.747387 2668 leader_election.cc:288] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:19.767546 2416 raft_consensus.cc:1261] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:19.767611 2556 raft_consensus.cc:1261] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:19.767843 2631 consensus_queue.cc:1035] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:19.768095 2631 consensus_queue.cc:1035] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 23:06:19.773234 2653 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 23:06:21.294382 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:21.294487 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:21.295367 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Sending DeleteTablet for 3 replicas of tablet f7cd9d7b20974985a8eaf83d8969810d
I20260924 23:06:21.295674 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd9d7b20974985a8eaf83d8969810d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55694
I20260924 23:06:21.295655 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd9d7b20974985a8eaf83d8969810d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55922
I20260924 23:06:21.295843 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet f7cd9d7b20974985a8eaf83d8969810d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39108
I20260924 23:06:21.296030 2705 tablet_replica.cc:330] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: stopping tablet replica
I20260924 23:06:21.296085 2705 raft_consensus.cc:2229] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:21.296120 2705 raft_consensus.cc:2258] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:21.296365 2705 ts_tablet_manager.cc:1905] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:21.297786 2704 tablet_replica.cc:330] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: stopping tablet replica
I20260924 23:06:21.297940 2704 raft_consensus.cc:2229] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:21.298044 2704 raft_consensus.cc:2258] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:21.298200 2703 tablet_replica.cc:330] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: stopping tablet replica
I20260924 23:06:21.298233 2705 ts_tablet_manager.cc:1918] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:06:21.298290 2705 log.cc:1196] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/f7cd9d7b20974985a8eaf83d8969810d
I20260924 23:06:21.298391 2703 raft_consensus.cc:2229] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:21.298508 2703 raft_consensus.cc:2258] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:21.298535 2705 ts_tablet_manager.cc:1939] T f7cd9d7b20974985a8eaf83d8969810d P 9673028ac2c34773a3750f25e30f4a96: Deleting consensus metadata
I20260924 23:06:21.298856 2704 ts_tablet_manager.cc:1905] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:21.298954 2106 catalog_manager.cc:4841] TS 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331): tablet f7cd9d7b20974985a8eaf83d8969810d (table test-table [id=63d48c59db3547a8bd649673bfcbb4f8]) successfully deleted
I20260924 23:06:21.300164 2703 ts_tablet_manager.cc:1905] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:21.300187 2704 ts_tablet_manager.cc:1918] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:06:21.300236 2704 log.cc:1196] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/f7cd9d7b20974985a8eaf83d8969810d
I20260924 23:06:21.300463 2704 ts_tablet_manager.cc:1939] T f7cd9d7b20974985a8eaf83d8969810d P 6c33b0b3e821435a835cbbec6b009098: Deleting consensus metadata
I20260924 23:06:21.300956 2104 catalog_manager.cc:4841] TS 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549): tablet f7cd9d7b20974985a8eaf83d8969810d (table test-table [id=63d48c59db3547a8bd649673bfcbb4f8]) successfully deleted
I20260924 23:06:21.301826 2703 ts_tablet_manager.cc:1918] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:06:21.301898 2703 log.cc:1196] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/f7cd9d7b20974985a8eaf83d8969810d
I20260924 23:06:21.302160 2703 ts_tablet_manager.cc:1939] T f7cd9d7b20974985a8eaf83d8969810d P 0ad35880300f492cab7fd0f27e034013: Deleting consensus metadata
I20260924 23:06:21.302606 2105 catalog_manager.cc:4841] TS 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845): tablet f7cd9d7b20974985a8eaf83d8969810d (table test-table [id=63d48c59db3547a8bd649673bfcbb4f8]) successfully deleted
Sep 24 23:06:21 dist-test-slave-ph9j krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:06:21 dist-test-slave-ph9j krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291181, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:21.323874 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:21.316180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.0.98.62:34241)
0924 23:06:21.316349 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:21.316350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:21.316367 (+ 17us) server_negotiation.cc:407] Connection header received
0924 23:06:21.316445 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:21.316446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:21.316456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:21.316523 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:21.317208 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:21.317770 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:21.318665 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:21.318881 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:21.318948 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:21.321308 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:21.321309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:21.321310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:21.321316 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:21.322907 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:21.323210 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:21.323211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:21.323211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:21.323261 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:21.323441 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:21.323442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:21.323442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:21.323544 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:21.323617 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:21.323703 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 23:06:21.323763 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
I20260924 23:06:21.325598 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:21.325754 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:21.327124 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 7554c4fd45ad40799329ff04e0dc9697 (DEFAULT_TABLE table=test-table [id=57103a37132247c1a0ad0cdebc25105f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:21.327184 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 7554c4fd45ad40799329ff04e0dc9697 (DEFAULT_TABLE table=test-table [id=57103a37132247c1a0ad0cdebc25105f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:21.327179 2536 tablet_service.cc:1467] Processing CreateTablet for tablet 7554c4fd45ad40799329ff04e0dc9697 (DEFAULT_TABLE table=test-table [id=57103a37132247c1a0ad0cdebc25105f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:21.327281 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7554c4fd45ad40799329ff04e0dc9697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:21.327292 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7554c4fd45ad40799329ff04e0dc9697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:21.327322 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7554c4fd45ad40799329ff04e0dc9697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:21.328673 2716 tablet_bootstrap.cc:492] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Bootstrap starting.
I20260924 23:06:21.328701 2715 tablet_bootstrap.cc:492] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Bootstrap starting.
I20260924 23:06:21.328810 2714 tablet_bootstrap.cc:492] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Bootstrap starting.
I20260924 23:06:21.329103 2716 tablet_bootstrap.cc:654] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:21.329116 2715 tablet_bootstrap.cc:654] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:21.329200 2714 tablet_bootstrap.cc:654] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:21.329734 2715 tablet_bootstrap.cc:492] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: No bootstrap required, opened a new log
I20260924 23:06:21.329787 2715 ts_tablet_manager.cc:1397] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:21.329902 2716 tablet_bootstrap.cc:492] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: No bootstrap required, opened a new log
I20260924 23:06:21.329905 2714 tablet_bootstrap.cc:492] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: No bootstrap required, opened a new log
I20260924 23:06:21.329957 2716 ts_tablet_manager.cc:1397] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:21.329957 2714 ts_tablet_manager.cc:1397] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:21.329941 2715 raft_consensus.cc:348] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.330025 2715 raft_consensus.cc:374] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:21.330050 2715 raft_consensus.cc:729] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Initialized, Role: FOLLOWER
I20260924 23:06:21.330075 2716 raft_consensus.cc:348] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.330191 2716 raft_consensus.cc:374] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:21.330152 2715 consensus_queue.cc:260] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.330286 2715 ts_tablet_manager.cc:1428] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:21.330220 2716 raft_consensus.cc:729] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9673028ac2c34773a3750f25e30f4a96, State: Initialized, Role: FOLLOWER
I20260924 23:06:21.330451 2716 consensus_queue.cc:260] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.330717 2716 ts_tablet_manager.cc:1428] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:21.331159 2714 raft_consensus.cc:348] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.331257 2714 raft_consensus.cc:374] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:21.331292 2714 raft_consensus.cc:729] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c33b0b3e821435a835cbbec6b009098, State: Initialized, Role: FOLLOWER
I20260924 23:06:21.331344 2714 consensus_queue.cc:260] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.331524 2714 ts_tablet_manager.cc:1428] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:21.335371 2697 raft_consensus.cc:482] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:21.335474 2697 raft_consensus.cc:504] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.335635 2697 leader_election.cc:288] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:21.335843 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7554c4fd45ad40799329ff04e0dc9697" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098" is_pre_election: true
I20260924 23:06:21.335812 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7554c4fd45ad40799329ff04e0dc9697" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96" is_pre_election: true
I20260924 23:06:21.335928 2556 raft_consensus.cc:2454] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:21.335928 2416 raft_consensus.cc:2454] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:21.336073 2208 leader_election.cc:302] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 6c33b0b3e821435a835cbbec6b009098; no voters:
I20260924 23:06:21.336163 2697 raft_consensus.cc:2781] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:21.336205 2697 raft_consensus.cc:482] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:21.336226 2697 raft_consensus.cc:3037] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:21.336894 2697 raft_consensus.cc:504] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.337034 2697 leader_election.cc:288] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Requested vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:21.337186 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7554c4fd45ad40799329ff04e0dc9697" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098"
I20260924 23:06:21.337227 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7554c4fd45ad40799329ff04e0dc9697" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96"
I20260924 23:06:21.337291 2556 raft_consensus.cc:3037] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:21.337311 2416 raft_consensus.cc:3037] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:21.337867 2416 raft_consensus.cc:2454] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:21.338121 2556 raft_consensus.cc:2454] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:21.338130 2210 leader_election.cc:302] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:21.338225 2697 raft_consensus.cc:2781] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:21.338270 2697 raft_consensus.cc:686] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Running, Role: LEADER
I20260924 23:06:21.338325 2697 consensus_queue.cc:237] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.338953 2118 catalog_manager.cc:5495] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ad35880300f492cab7fd0f27e034013 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ad35880300f492cab7fd0f27e034013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:21.350092 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:21.347505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.0.98.1:41845)
0924 23:06:21.347648 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:21.347650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:21.347655 (+ 5us) server_negotiation.cc:407] Connection header received
0924 23:06:21.347673 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:21.347674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:21.347683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:21.347759 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:21.347899 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:21.348379 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:21.349294 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:21.349482 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:21.349549 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:21.349742 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:21.349774 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:21.349827 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:21.349979 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 23:06:21.350001 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260924 23:06:21.352589 2556 raft_consensus.cc:1261] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:21.352632 2416 raft_consensus.cc:1261] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:21.352947 2697 consensus_queue.cc:1035] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:21.353050 2681 consensus_queue.cc:1035] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:21.363090 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:21.363197 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:21.363835 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Sending DeleteTablet for 3 replicas of tablet 7554c4fd45ad40799329ff04e0dc9697
I20260924 23:06:21.364071 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 7554c4fd45ad40799329ff04e0dc9697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55694
I20260924 23:06:21.364126 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet 7554c4fd45ad40799329ff04e0dc9697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55922
I20260924 23:06:21.364198 2703 tablet_replica.cc:330] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: stopping tablet replica
I20260924 23:06:21.364233 2704 tablet_replica.cc:330] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: stopping tablet replica
I20260924 23:06:21.364235 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 7554c4fd45ad40799329ff04e0dc9697 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39108
I20260924 23:06:21.364283 2704 raft_consensus.cc:2229] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:21.364279 2703 raft_consensus.cc:2229] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:21.364327 2704 raft_consensus.cc:2258] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:21.364338 2705 tablet_replica.cc:330] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: stopping tablet replica
I20260924 23:06:21.364353 2703 raft_consensus.cc:2258] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:21.364411 2705 raft_consensus.cc:2229] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:21.364451 2705 raft_consensus.cc:2258] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:21.364661 2704 ts_tablet_manager.cc:1905] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:21.364621 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:21.364763 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:21.364890 2703 ts_tablet_manager.cc:1905] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:21.365958 2256 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd197320f34827ad264237d4857927 (DEFAULT_TABLE table=test-table [id=4f6ba923bf444df09875c1b6c6a1bdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:21.366065 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd197320f34827ad264237d4857927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:21.366060 2536 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd197320f34827ad264237d4857927 (DEFAULT_TABLE table=test-table [id=4f6ba923bf444df09875c1b6c6a1bdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:21.366154 2536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd197320f34827ad264237d4857927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:21.366477 2704 ts_tablet_manager.cc:1918] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:21.366528 2704 log.cc:1196] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/7554c4fd45ad40799329ff04e0dc9697
I20260924 23:06:21.366680 2703 ts_tablet_manager.cc:1918] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:21.366734 2703 log.cc:1196] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/7554c4fd45ad40799329ff04e0dc9697
I20260924 23:06:21.366741 2704 ts_tablet_manager.cc:1939] T 7554c4fd45ad40799329ff04e0dc9697 P 6c33b0b3e821435a835cbbec6b009098: Deleting consensus metadata
I20260924 23:06:21.366966 2703 ts_tablet_manager.cc:1939] T 7554c4fd45ad40799329ff04e0dc9697 P 0ad35880300f492cab7fd0f27e034013: Deleting consensus metadata
I20260924 23:06:21.367051 2705 ts_tablet_manager.cc:1905] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:21.367277 2105 catalog_manager.cc:4841] TS 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845): tablet 7554c4fd45ad40799329ff04e0dc9697 (table test-table [id=57103a37132247c1a0ad0cdebc25105f]) successfully deleted
I20260924 23:06:21.367486 2715 tablet_bootstrap.cc:492] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013: Bootstrap starting.
I20260924 23:06:21.367798 2715 tablet_bootstrap.cc:654] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:21.367794 2396 tablet_service.cc:1467] Processing CreateTablet for tablet e6bd197320f34827ad264237d4857927 (DEFAULT_TABLE table=test-table [id=4f6ba923bf444df09875c1b6c6a1bdec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:21.367960 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bd197320f34827ad264237d4857927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:21.368157 2714 tablet_bootstrap.cc:492] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Bootstrap starting.
I20260924 23:06:21.368356 2715 tablet_bootstrap.cc:492] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013: No bootstrap required, opened a new log
I20260924 23:06:21.368391 2715 ts_tablet_manager.cc:1397] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:21.368495 2715 raft_consensus.cc:348] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.368556 2715 raft_consensus.cc:374] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:21.368573 2715 raft_consensus.cc:729] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Initialized, Role: FOLLOWER
I20260924 23:06:21.368621 2715 consensus_queue.cc:260] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.368722 2715 ts_tablet_manager.cc:1428] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:21.368765 2705 ts_tablet_manager.cc:1918] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:21.368819 2705 log.cc:1196] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/7554c4fd45ad40799329ff04e0dc9697
I20260924 23:06:21.369043 2705 ts_tablet_manager.cc:1939] T 7554c4fd45ad40799329ff04e0dc9697 P 9673028ac2c34773a3750f25e30f4a96: Deleting consensus metadata
I20260924 23:06:21.369056 2714 tablet_bootstrap.cc:654] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:21.369414 2716 tablet_bootstrap.cc:492] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Bootstrap starting.
I20260924 23:06:21.369491 2104 catalog_manager.cc:4841] TS 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549): tablet 7554c4fd45ad40799329ff04e0dc9697 (table test-table [id=57103a37132247c1a0ad0cdebc25105f]) successfully deleted
I20260924 23:06:21.369717 2716 tablet_bootstrap.cc:654] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:21.369840 2106 catalog_manager.cc:4841] TS 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331): tablet 7554c4fd45ad40799329ff04e0dc9697 (table test-table [id=57103a37132247c1a0ad0cdebc25105f]) successfully deleted
I20260924 23:06:21.370028 2714 tablet_bootstrap.cc:492] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: No bootstrap required, opened a new log
I20260924 23:06:21.370069 2714 ts_tablet_manager.cc:1397] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:21.370273 2716 tablet_bootstrap.cc:492] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: No bootstrap required, opened a new log
I20260924 23:06:21.370226 2714 raft_consensus.cc:348] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.370321 2714 raft_consensus.cc:374] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:21.370321 2716 ts_tablet_manager.cc:1397] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:21.370353 2714 raft_consensus.cc:729] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c33b0b3e821435a835cbbec6b009098, State: Initialized, Role: FOLLOWER
I20260924 23:06:21.370437 2716 raft_consensus.cc:348] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.370508 2716 raft_consensus.cc:374] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:21.370529 2716 raft_consensus.cc:729] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9673028ac2c34773a3750f25e30f4a96, State: Initialized, Role: FOLLOWER
I20260924 23:06:21.370491 2714 consensus_queue.cc:260] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.370573 2716 consensus_queue.cc:260] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.370597 2714 ts_tablet_manager.cc:1428] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:21.370671 2716 ts_tablet_manager.cc:1428] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:21.471128 2694 raft_consensus.cc:482] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:21.471231 2694 raft_consensus.cc:504] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.471455 2694 leader_election.cc:288] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:21.471697 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd197320f34827ad264237d4857927" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098" is_pre_election: true
I20260924 23:06:21.471704 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd197320f34827ad264237d4857927" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96" is_pre_election: true
I20260924 23:06:21.471801 2556 raft_consensus.cc:2454] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:21.471801 2416 raft_consensus.cc:2454] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 0.
I20260924 23:06:21.472025 2210 leader_election.cc:302] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 9673028ac2c34773a3750f25e30f4a96; no voters:
I20260924 23:06:21.472143 2694 raft_consensus.cc:2781] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:21.472193 2694 raft_consensus.cc:482] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:21.472216 2694 raft_consensus.cc:3037] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:21.472935 2694 raft_consensus.cc:504] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.473080 2694 leader_election.cc:288] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Requested vote from peers 9673028ac2c34773a3750f25e30f4a96 (127.0.98.2:40331), 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549)
I20260924 23:06:21.473210 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd197320f34827ad264237d4857927" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c33b0b3e821435a835cbbec6b009098"
I20260924 23:06:21.473306 2556 raft_consensus.cc:3037] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:21.473312 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bd197320f34827ad264237d4857927" candidate_uuid: "0ad35880300f492cab7fd0f27e034013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9673028ac2c34773a3750f25e30f4a96"
I20260924 23:06:21.473388 2416 raft_consensus.cc:3037] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:21.474092 2556 raft_consensus.cc:2454] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:21.474169 2416 raft_consensus.cc:2454] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ad35880300f492cab7fd0f27e034013 in term 1.
I20260924 23:06:21.474304 2208 leader_election.cc:302] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad35880300f492cab7fd0f27e034013, 6c33b0b3e821435a835cbbec6b009098; no voters:
I20260924 23:06:21.474404 2694 raft_consensus.cc:2781] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:21.474491 2694 raft_consensus.cc:686] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad35880300f492cab7fd0f27e034013, State: Running, Role: LEADER
I20260924 23:06:21.474551 2694 consensus_queue.cc:237] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } }
I20260924 23:06:21.475170 2118 catalog_manager.cc:5495] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ad35880300f492cab7fd0f27e034013 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0ad35880300f492cab7fd0f27e034013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad35880300f492cab7fd0f27e034013" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:21.489055 2556 raft_consensus.cc:1261] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:21.489144 2416 raft_consensus.cc:1261] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Refusing update from remote peer 0ad35880300f492cab7fd0f27e034013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:21.489275 2694 consensus_queue.cc:1035] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c33b0b3e821435a835cbbec6b009098" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:21.489372 2681 consensus_queue.cc:1035] T e6bd197320f34827ad264237d4857927 P 0ad35880300f492cab7fd0f27e034013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9673028ac2c34773a3750f25e30f4a96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 23:06:21.493742 2687 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 23:06:23.010525 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:23.010648 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:23.011550 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a51d1c89c4d40ccb9c46ce132e03f05: Sending DeleteTablet for 3 replicas of tablet e6bd197320f34827ad264237d4857927
I20260924 23:06:23.011807 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet e6bd197320f34827ad264237d4857927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39108
I20260924 23:06:23.011861 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet e6bd197320f34827ad264237d4857927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55694
I20260924 23:06:23.011984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 2191
I20260924 23:06:23.012521 2536 tablet_service.cc:1514] Processing DeleteTablet for tablet e6bd197320f34827ad264237d4857927 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55922
I20260924 23:06:23.012881 2756 tablet_replica.cc:330] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: stopping tablet replica
I20260924 23:06:23.012979 2756 raft_consensus.cc:2229] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:23.013034 2756 raft_consensus.cc:2258] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:23.013276 2756 ts_tablet_manager.cc:1905] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:23.015123 2756 ts_tablet_manager.cc:1918] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:06:23.015210 2756 log.cc:1196] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/e6bd197320f34827ad264237d4857927
I20260924 23:06:23.015497 2756 ts_tablet_manager.cc:1939] T e6bd197320f34827ad264237d4857927 P 6c33b0b3e821435a835cbbec6b009098: Deleting consensus metadata
I20260924 23:06:23.015903 2104 catalog_manager.cc:4841] TS 6c33b0b3e821435a835cbbec6b009098 (127.0.98.3:35549): tablet e6bd197320f34827ad264237d4857927 (table test-table [id=4f6ba923bf444df09875c1b6c6a1bdec]) successfully deleted
I20260924 23:06:23.018013 2754 tablet_replica.cc:330] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: stopping tablet replica
I20260924 23:06:23.018100 2754 raft_consensus.cc:2229] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:23.018172 2754 raft_consensus.cc:2258] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:23.018422 2754 ts_tablet_manager.cc:1905] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 23:06:23.019469 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41845 (error 108)
I20260924 23:06:23.020076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 2331
I20260924 23:06:23.020161 2754 ts_tablet_manager.cc:1918] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 23:06:23.020220 2754 log.cc:1196] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/e6bd197320f34827ad264237d4857927
I20260924 23:06:23.020431 2754 ts_tablet_manager.cc:1939] T e6bd197320f34827ad264237d4857927 P 9673028ac2c34773a3750f25e30f4a96: Deleting consensus metadata
I20260924 23:06:23.021476 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:23.019906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41845 (local address 127.0.0.1:55724)
0924 23:06:23.021366 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:23.021426 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1431,"thread_start_us":76,"threads_started":1}
W20260924 23:06:23.021646 2105 catalog_manager.cc:4584] TS 0ad35880300f492cab7fd0f27e034013 (127.0.98.1:41845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6bd197320f34827ad264237d4857927: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41845: connect: Connection refused (error 111)
I20260924 23:06:23.026145 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:23.026023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40331 (local address 127.0.0.1:39120)
0924 23:06:23.026078 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:23.026105 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:06:23.026134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 2471
I20260924 23:06:23.030781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 2087
2026-09-24T23:06:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5423 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:23 dist-test-slave-ph9j krb5kdc[2761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:23 dist-test-slave-ph9j krb5kdc[2761](info): set up 2 sockets
Sep 24 23:06:23 dist-test-slave-ph9j krb5kdc[2761](info): commencing operation
krb5kdc: starting...
W20260924 23:06:24.117704 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:24.236145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43751
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:24.315430 2774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:24.315690 2774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:24.315737 2774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:24.316915 2774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:24.316993 2774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:24.317016 2774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:24.317037 2774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:24.317068 2774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:24.318625 2774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:24.318917 2774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:24.319151 2774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:24.324225 2782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:24.324201 2780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.324342 2774 server_base.cc:1034] running on GCE node
W20260924 23:06:24.324185 2779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.324648 2774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:24.324975 2774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:24.326144 2774 hybrid_clock.cc:648] HybridClock initialized: now 1790291184326134 us; error 40 us; skew 500 ppm
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:24.328594 2774 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:24.329046 2774 webserver.cc:460] Webserver started at http://127.0.98.62:42933/ using document root <none> and password file <none>
I20260924 23:06:24.329288 2774 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:24.329360 2774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:24.329520 2774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:24.330684 2774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "86db8a466a2540e9bc711f8d9b001c84"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "2a895f3e1eb0f38a3273ce5ec9348716"
server_key_iv: "3fe42c71e32f02239d7c2c328929ff97"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.331050 2774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "86db8a466a2540e9bc711f8d9b001c84"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "2a895f3e1eb0f38a3273ce5ec9348716"
server_key_iv: "3fe42c71e32f02239d7c2c328929ff97"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.332336 2774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:24.333101 2789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.333321 2774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:24.333431 2774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "86db8a466a2540e9bc711f8d9b001c84"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "2a895f3e1eb0f38a3273ce5ec9348716"
server_key_iv: "3fe42c71e32f02239d7c2c328929ff97"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.333509 2774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:24.349927 2774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:24.351891 2774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:24.352077 2774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:24.356249 2774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43751
I20260924 23:06:24.356284 2841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43751 every 8 connection(s)
I20260924 23:06:24.356799 2774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:24.357510 2842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:24.359782 2842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84: Bootstrap starting.
I20260924 23:06:24.360354 2842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:24.360644 2842 log.cc:824] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:24.360704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 2774
I20260924 23:06:24.360811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:24.361014 392 external_mini_cluster.cc:1428] Setting key 00a37514349ad9a01859e474e31ead3c
I20260924 23:06:24.361487 2842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84: No bootstrap required, opened a new log
I20260924 23:06:24.364107 2842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db8a466a2540e9bc711f8d9b001c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43751 } }
I20260924 23:06:24.364269 2842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:24.364298 2842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86db8a466a2540e9bc711f8d9b001c84, State: Initialized, Role: FOLLOWER
I20260924 23:06:24.364409 2842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db8a466a2540e9bc711f8d9b001c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43751 } }
I20260924 23:06:24.364480 2842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:24.364507 2842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:24.364548 2842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:24.365162 2842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db8a466a2540e9bc711f8d9b001c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43751 } }
I20260924 23:06:24.365247 2842 leader_election.cc:302] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86db8a466a2540e9bc711f8d9b001c84; no voters:
I20260924 23:06:24.365424 2842 leader_election.cc:288] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:24.365445 2847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:24.365551 2847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [term 1 LEADER]: Becoming Leader. State: Replica: 86db8a466a2540e9bc711f8d9b001c84, State: Running, Role: LEADER
I20260924 23:06:24.365630 2847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db8a466a2540e9bc711f8d9b001c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43751 } }
I20260924 23:06:24.365679 2842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:24.365953 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86db8a466a2540e9bc711f8d9b001c84. Latest consensus state: current_term: 1 leader_uuid: "86db8a466a2540e9bc711f8d9b001c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db8a466a2540e9bc711f8d9b001c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43751 } } }
I20260924 23:06:24.366170 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:24.366114 2849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86db8a466a2540e9bc711f8d9b001c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86db8a466a2540e9bc711f8d9b001c84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43751 } } }
I20260924 23:06:24.366561 2849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:24.366745 2855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:24.367264 2855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:24.368795 2855 catalog_manager.cc:1347] Generated new cluster ID: bd17229f83d745408c730c45f44ba442
I20260924 23:06:24.368846 2855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:24.372637 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.361527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.0.98.62:43751)
0924 23:06:24.361687 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:24.361698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:24.361732 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:06:24.362715 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:24.362724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:24.362787 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:24.362939 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:24.363520 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.364870 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.365834 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.366531 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.366636 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.369482 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:24.369486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:24.369487 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:24.369505 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:24.371184 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.371551 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.371554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.371554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.371641 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.371855 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.371856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.371856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.372078 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:24.372232 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:24.372337 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 23:06:24.372479 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260924 23:06:24.388417 2855 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:24.389012 2855 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:24.393996 2855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86db8a466a2540e9bc711f8d9b001c84: Generated new TSK 0
I20260924 23:06:24.394207 2855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:24.428668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43751
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:24.516448 2868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:24.516688 2868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:24.516731 2868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:24.517928 2868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:24.517997 2868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:24.518045 2868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:24.519564 2868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:24.519827 2868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:24.520061 2868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:24.529485 2876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:24.529520 2874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:24.529485 2873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.530826 2868 server_base.cc:1034] running on GCE node
I20260924 23:06:24.531097 2868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:24.531383 2868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:24.532562 2868 hybrid_clock.cc:648] HybridClock initialized: now 1790291184532551 us; error 36 us; skew 500 ppm
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:24.535403 2868 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:24.535876 2868 webserver.cc:460] Webserver started at http://127.0.98.1:40605/ using document root <none> and password file <none>
I20260924 23:06:24.536149 2868 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:24.536211 2868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:24.536371 2868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:24.537518 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "62f2d1e123634bd7a691cefdd61f5baa"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "95c254d02a5818a8b3476e4de7550c36"
server_key_iv: "037b439fd1f9642391ecc8ca0d0fc14f"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.537927 2868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62f2d1e123634bd7a691cefdd61f5baa"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "95c254d02a5818a8b3476e4de7550c36"
server_key_iv: "037b439fd1f9642391ecc8ca0d0fc14f"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.539462 2868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:24.540208 2883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.540383 2868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.540488 2868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "62f2d1e123634bd7a691cefdd61f5baa"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "95c254d02a5818a8b3476e4de7550c36"
server_key_iv: "037b439fd1f9642391ecc8ca0d0fc14f"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.540575 2868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:24.566952 2868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:24.569469 2868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:24.569645 2868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:24.569986 2868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:24.570331 2868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:24.570410 2868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.570446 2868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:24.570474 2868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.576273 2868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39427
I20260924 23:06:24.576385 2996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39427 every 8 connection(s)
I20260924 23:06:24.576699 2868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:24.584530 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 2868
I20260924 23:06:24.584635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:24.584815 392 external_mini_cluster.cc:1428] Setting key bfe87efa00723282996d4467cd7f261c
I20260924 23:06:24.586524 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.577493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33147 (local address 127.0.98.62:43751)
0924 23:06:24.577547 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:24.577550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:24.577773 (+ 223us) server_negotiation.cc:407] Connection header received
0924 23:06:24.578890 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:24.578891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:24.578902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:24.578948 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:24.579545 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.580042 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.581002 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.581202 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.581257 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.583485 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:24.583487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:24.583487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:24.583494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:24.585055 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.585473 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.585474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.585474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.585534 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.585800 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.585801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.585801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.585985 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:24.586060 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:24.586333 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 23:06:24.586425 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 23:06:24.586725 2999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.577479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43751 (local address 127.0.98.1:33147)
0924 23:06:24.577636 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:24.577674 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:24.578774 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:24.579014 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:24.579017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:24.579270 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:24.579473 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:24.579476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.580123 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.580124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:24.580925 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:24.580926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.581280 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.581281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:24.581400 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.582026 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:24.582034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:24.583386 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:24.585155 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:24.585164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:24.585166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:24.585388 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:24.585609 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:24.585610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:24.585610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:24.585723 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:24.586051 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:24.586057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:24.586198 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 23:06:24.586312 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:06:24.586476 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260924 23:06:24.587324 2997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43751
I20260924 23:06:24.587440 2997 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:24.587643 2997 heartbeater.cc:511] Master 127.0.98.62:43751 requested a full tablet report, sending...
I20260924 23:06:24.588225 2806 ts_manager.cc:194] Registered new tserver with Master: 62f2d1e123634bd7a691cefdd61f5baa (127.0.98.1:39427)
I20260924 23:06:24.588860 2806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33147
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:24.608994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43751
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:24.689689 3002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:24.689983 3002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:24.690014 3002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:24.691183 3002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:24.691224 3002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:24.691251 3002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:24.692776 3002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:24.693033 3002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:24.693264 3002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:24.698233 3007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:24.698220 3008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.698557 3002 server_base.cc:1034] running on GCE node
W20260924 23:06:24.698220 3010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.698848 3002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:24.699115 3002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:24.700253 3002 hybrid_clock.cc:648] HybridClock initialized: now 1790291184700232 us; error 37 us; skew 500 ppm
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:24.702831 3002 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:24.703300 3002 webserver.cc:460] Webserver started at http://127.0.98.2:44575/ using document root <none> and password file <none>
I20260924 23:06:24.703541 3002 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:24.703608 3002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:24.703763 3002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:24.704813 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "7d5627f417ae4b1abf0dcf280117e2de"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "715c30f4efcad7b39bd7fd839a64e139"
server_key_iv: "c4995480da6af89fe925a091cfbe3775"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.705183 3002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7d5627f417ae4b1abf0dcf280117e2de"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "715c30f4efcad7b39bd7fd839a64e139"
server_key_iv: "c4995480da6af89fe925a091cfbe3775"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.706573 3002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:24.707268 3017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.707456 3002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:24.707561 3002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "7d5627f417ae4b1abf0dcf280117e2de"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "715c30f4efcad7b39bd7fd839a64e139"
server_key_iv: "c4995480da6af89fe925a091cfbe3775"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.707638 3002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:24.714501 3002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:24.716729 3002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:24.716887 3002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:24.717105 3002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:24.717396 3002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:24.717448 3002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.717489 3002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:24.717517 3002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.723325 3002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43055
I20260924 23:06:24.723364 3130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43055 every 8 connection(s)
I20260924 23:06:24.723667 3002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:24.732580 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.724355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51773 (local address 127.0.98.62:43751)
0924 23:06:24.724417 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:24.724419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:24.724590 (+ 171us) server_negotiation.cc:407] Connection header received
0924 23:06:24.725481 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:24.725482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:24.725491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:24.725552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:24.726188 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.726675 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.727596 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.727777 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.727828 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.730143 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:24.730144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:24.730144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:24.730151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:24.731434 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.731800 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.731801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.731801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.731853 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.732063 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.732064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.732064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.732178 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:24.732236 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:24.732451 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 23:06:24.732514 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:06:24.732758 3133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.724318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43751 (local address 127.0.98.2:51773)
0924 23:06:24.724467 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:24.724506 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:24.725398 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:24.725605 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:24.725608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:24.725841 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:24.726094 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:24.726099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.726753 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.726753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:24.727526 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:24.727527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.727846 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.727848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:24.727941 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.728462 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:24.728470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:24.730058 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:24.731506 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:24.731510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:24.731512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:24.731717 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:24.731913 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:24.731913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:24.731913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:24.732006 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:24.732234 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:24.732237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:24.732342 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 23:06:24.732422 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:06:24.732533 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260924 23:06:24.733247 3131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43751
I20260924 23:06:24.733325 3131 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:24.733510 3131 heartbeater.cc:511] Master 127.0.98.62:43751 requested a full tablet report, sending...
I20260924 23:06:24.733574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3002
I20260924 23:06:24.733690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:24.733939 392 external_mini_cluster.cc:1428] Setting key 5b761adec5e0fd99b1fdd7a9b04ecb13
I20260924 23:06:24.733922 2806 ts_manager.cc:194] Registered new tserver with Master: 7d5627f417ae4b1abf0dcf280117e2de (127.0.98.2:43055)
I20260924 23:06:24.734483 2806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51773
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:24.758679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43751
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:24.843114 3136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:24.843331 3136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:24.843370 3136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:24.845094 3136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:24.845177 3136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:24.845220 3136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:24.846756 3136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43751
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:24.847023 3136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:24.847239 3136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:24.852452 3144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:24.852474 3142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.852541 3136 server_base.cc:1034] running on GCE node
W20260924 23:06:24.852650 3141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:24.852857 3136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:24.853111 3136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:24.854269 3136 hybrid_clock.cc:648] HybridClock initialized: now 1790291184854258 us; error 45 us; skew 500 ppm
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:24.856580 3136 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:24.856993 3136 webserver.cc:460] Webserver started at http://127.0.98.3:40815/ using document root <none> and password file <none>
I20260924 23:06:24.857219 3136 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:24.857285 3136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:24.857432 3136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:24.858588 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a5fc2c8dac5406982c3337490f92a10"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "72c76fa3dbcbd294989538f98972b2f3"
server_key_iv: "29ad8160f590e181020cb6d186795781"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.858945 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a5fc2c8dac5406982c3337490f92a10"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "72c76fa3dbcbd294989538f98972b2f3"
server_key_iv: "29ad8160f590e181020cb6d186795781"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.860244 3136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:24.860967 3151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.861141 3136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:24.861258 3136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "9a5fc2c8dac5406982c3337490f92a10"
format_stamp: "Formatted at 2026-09-24 23:06:24 on dist-test-slave-ph9j"
server_key: "72c76fa3dbcbd294989538f98972b2f3"
server_key_iv: "29ad8160f590e181020cb6d186795781"
server_key_version: "encryptionkey@0"
I20260924 23:06:24.861323 3136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:24.874109 3136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:24.875943 3136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:24.876060 3136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:24.876250 3136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:24.876515 3136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:24.876549 3136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.876572 3136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:24.876585 3136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:24.882722 3136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42305
I20260924 23:06:24.882818 3264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42305 every 8 connection(s)
I20260924 23:06:24.883122 3136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:24.883535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3136
I20260924 23:06:24.883631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:24.883846 392 external_mini_cluster.cc:1428] Setting key 58ed4589f1e1f8beb2bf12d3a35898d9
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[2761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:24.892541 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.883925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38027 (local address 127.0.98.62:43751)
0924 23:06:24.883980 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:24.883983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:24.884152 (+ 169us) server_negotiation.cc:407] Connection header received
0924 23:06:24.885210 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:24.885211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:24.885223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:24.885267 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:24.885912 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.886419 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.887428 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.887637 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.887721 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.889850 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:24.889851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:24.889851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:24.889858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:24.891237 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.891624 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.891624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.891625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.891674 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:24.891925 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:24.891925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:24.891926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:24.892027 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:24.892113 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:24.892408 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 23:06:24.892476 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:06:24.892807 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:24.883830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43751 (local address 127.0.98.3:38027)
0924 23:06:24.884018 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:24.884059 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:24.885109 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:24.885333 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:24.885337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:24.885601 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:24.885788 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:24.885792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.886527 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.886529 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:24.887336 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:24.887337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:24.887731 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:24.887732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:24.887861 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:24.888401 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:24.888410 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:24.889766 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:24.891318 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:24.891325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:24.891326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:24.891552 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:24.891759 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:24.891760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:24.891760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:24.891867 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:24.892150 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:24.892156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:24.892276 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 23:06:24.892375 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 23:06:24.892521 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260924 23:06:24.893337 3265 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43751
I20260924 23:06:24.893427 3265 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:24.893630 3265 heartbeater.cc:511] Master 127.0.98.62:43751 requested a full tablet report, sending...
I20260924 23:06:24.894080 2806 ts_manager.cc:194] Registered new tserver with Master: 9a5fc2c8dac5406982c3337490f92a10 (127.0.98.3:42305)
I20260924 23:06:24.894461 2806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38027
I20260924 23:06:24.895530 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 23:06:24.906560 2846 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:24.905558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.0.98.62:43751)
0924 23:06:24.905629 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:24.905631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:24.905771 (+ 140us) server_negotiation.cc:407] Connection header received
0924 23:06:24.905836 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:24.905837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:24.905846 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:24.905935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:24.906505 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33088: BlockingRecv error: recv got EOF from 127.0.0.1:33088 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:24.907380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 2868
I20260924 23:06:24.912153 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3002
I20260924 23:06:24.916602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3136
I20260924 23:06:24.920794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 2774
2026-09-24T23:06:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1889 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[3277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[3277](info): set up 2 sockets
Sep 24 23:06:24 dist-test-slave-ph9j krb5kdc[3277](info): commencing operation
krb5kdc: starting...
W20260924 23:06:25.961751 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:26 dist-test-slave-ph9j krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:26.081638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37595
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:26.161015 3290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:26.161264 3290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:26.161301 3290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:26.162453 3290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:26.162531 3290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:26.162555 3290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:26.162576 3290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:26.162606 3290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:26.164083 3290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:26.164357 3290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:26.164585 3290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:26.169557 3295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:26.169559 3296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:26.169745 3290 server_base.cc:1034] running on GCE node
W20260924 23:06:26.169596 3298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:26.170073 3290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:26.170356 3290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:26.171528 3290 hybrid_clock.cc:648] HybridClock initialized: now 1790291186171523 us; error 57 us; skew 500 ppm
Sep 24 23:06:26 dist-test-slave-ph9j krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:26.174114 3290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:26.174553 3290 webserver.cc:460] Webserver started at http://127.0.98.62:43817/ using document root <none> and password file <none>
I20260924 23:06:26.174747 3290 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:26.174787 3290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:26.174883 3290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:26.175876 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "9687b29028644cd29dce066b7d113f87"
format_stamp: "Formatted at 2026-09-24 23:06:26 on dist-test-slave-ph9j"
server_key: "7cea0cd327acc26bb52da7f7b1f2db87"
server_key_iv: "e1ec0f2f2d26c3b56720b4a96314899f"
server_key_version: "encryptionkey@0"
I20260924 23:06:26.176163 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "9687b29028644cd29dce066b7d113f87"
format_stamp: "Formatted at 2026-09-24 23:06:26 on dist-test-slave-ph9j"
server_key: "7cea0cd327acc26bb52da7f7b1f2db87"
server_key_iv: "e1ec0f2f2d26c3b56720b4a96314899f"
server_key_version: "encryptionkey@0"
I20260924 23:06:26.177366 3290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:26.178062 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:26.178220 3290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:26.178313 3290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "9687b29028644cd29dce066b7d113f87"
format_stamp: "Formatted at 2026-09-24 23:06:26 on dist-test-slave-ph9j"
server_key: "7cea0cd327acc26bb52da7f7b1f2db87"
server_key_iv: "e1ec0f2f2d26c3b56720b4a96314899f"
server_key_version: "encryptionkey@0"
I20260924 23:06:26.178368 3290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:26.197628 3290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:26.199836 3290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:26.199965 3290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:26.203866 3290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37595
I20260924 23:06:26.203923 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37595 every 8 connection(s)
I20260924 23:06:26.204285 3290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:26.205214 3358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:26.206460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3290
I20260924 23:06:26.206537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:26.206728 392 external_mini_cluster.cc:1428] Setting key 56c026f90d86e8419f078ddd9bd8f1ad
I20260924 23:06:26.208066 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87: Bootstrap starting.
I20260924 23:06:26.208809 3358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:26.209143 3358 log.cc:824] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:26.209851 3358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87: No bootstrap required, opened a new log
I20260924 23:06:26.212133 3358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9687b29028644cd29dce066b7d113f87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260924 23:06:26.212311 3358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:26.212352 3358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9687b29028644cd29dce066b7d113f87, State: Initialized, Role: FOLLOWER
I20260924 23:06:26.212491 3358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9687b29028644cd29dce066b7d113f87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260924 23:06:26.212579 3358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:26.212612 3358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:26.212668 3358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:26.213449 3358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9687b29028644cd29dce066b7d113f87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260924 23:06:26.213533 3358 leader_election.cc:302] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9687b29028644cd29dce066b7d113f87; no voters:
I20260924 23:06:26.213698 3358 leader_election.cc:288] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:26.213729 3363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:26.213848 3363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [term 1 LEADER]: Becoming Leader. State: Replica: 9687b29028644cd29dce066b7d113f87, State: Running, Role: LEADER
I20260924 23:06:26.213979 3358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 23:06:26 dist-test-slave-ph9j krb5kdc[3277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:26.213960 3363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9687b29028644cd29dce066b7d113f87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } }
I20260924 23:06:26.216109 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9687b29028644cd29dce066b7d113f87. Latest consensus state: current_term: 1 leader_uuid: "9687b29028644cd29dce066b7d113f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9687b29028644cd29dce066b7d113f87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } } }
I20260924 23:06:26.216454 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:26.216121 3364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9687b29028644cd29dce066b7d113f87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9687b29028644cd29dce066b7d113f87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37595 } } }
I20260924 23:06:26.216526 3364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87 [sys.catalog]: This master's current role is: LEADER
W20260924 23:06:26.216805 3377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:06:26.216859 3377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:06:26.216948 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:26.217177 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:26.218544 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:26.207255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.62:37595)
0924 23:06:26.207449 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:26.207459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:26.207494 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:06:26.208038 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:26.208046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:26.208108 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:26.208254 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:26.209199 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:26.210258 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:26.211317 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:26.211700 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:26.211783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:26.214660 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:26.214664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:26.214664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:26.214676 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:26.216819 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:26.217260 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:26.217263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:26.217264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:26.217364 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:26.217599 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:26.217600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:26.217601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:26.217930 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:26.218075 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:26.218185 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 23:06:26.218318 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260924 23:06:26.218824 3379 catalog_manager.cc:1347] Generated new cluster ID: 4bde242c8f914261b32d24b2d323429c
I20260924 23:06:26.218870 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:26.219311 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:06:26.225242 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:26.226006 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:26.231653 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9687b29028644cd29dce066b7d113f87: Generated new TSK 0
I20260924 23:06:26.231848 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:06:26.235024 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:26.231407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.0.98.62:37595)
0924 23:06:26.231445 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:26.231447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:26.231455 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:26.231478 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:26.231479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:26.231490 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:26.231547 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:26.231988 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:26.232472 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:26.233566 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:26.233755 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:26.233804 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:26.234540 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:26.234542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:26.234542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:26.234745 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:26.234761 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:26.234766 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:26.234811 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:26.234899 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:06:26.234923 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:06:26.237346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3290
2026-09-24T23:06:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1313 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:26 dist-test-slave-ph9j krb5kdc[3391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:26 dist-test-slave-ph9j krb5kdc[3391](info): set up 2 sockets
Sep 24 23:06:26 dist-test-slave-ph9j krb5kdc[3391](info): commencing operation
krb5kdc: starting...
W20260924 23:06:27.274677 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:27.395017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39405
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:27.475246 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:27.475467 3404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:27.475490 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:27.476593 3404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:27.476629 3404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:27.476643 3404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:27.476656 3404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:27.476668 3404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:27.478116 3404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:27.478403 3404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:27.478672 3404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:27.484615 3410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:27.484583 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:27.484797 3404 server_base.cc:1034] running on GCE node
W20260924 23:06:27.484884 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:27.485206 3404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:27.485582 3404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:27.486768 3404 hybrid_clock.cc:648] HybridClock initialized: now 1790291187486734 us; error 48 us; skew 500 ppm
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:27.490301 3404 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:27.490825 3404 webserver.cc:460] Webserver started at http://127.0.98.62:43613/ using document root <none> and password file <none>
I20260924 23:06:27.491134 3404 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:27.491206 3404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:27.491371 3404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:27.492539 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "1e337447ef1e4c8887e11a685d27ecaa"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "09f3ba18bc3888ff2f9742d16988645b"
server_key_iv: "b30e8035956c3347922fe6dd48eaf5be"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.492995 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e337447ef1e4c8887e11a685d27ecaa"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "09f3ba18bc3888ff2f9742d16988645b"
server_key_iv: "b30e8035956c3347922fe6dd48eaf5be"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.494541 3404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:27.495384 3419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.495606 3404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:27.495720 3404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "1e337447ef1e4c8887e11a685d27ecaa"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "09f3ba18bc3888ff2f9742d16988645b"
server_key_iv: "b30e8035956c3347922fe6dd48eaf5be"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.495824 3404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:27.508528 3404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:27.511500 3404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:27.511703 3404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:27.516149 3404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39405
I20260924 23:06:27.516178 3471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39405 every 8 connection(s)
I20260924 23:06:27.516618 3404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:27.517604 3472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:27.519896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3404
I20260924 23:06:27.519997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:27.520092 3472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa: Bootstrap starting.
I20260924 23:06:27.520172 392 external_mini_cluster.cc:1428] Setting key 23d990329612a2d505bd68fb43a24e71
I20260924 23:06:27.520807 3472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:27.521098 3472 log.cc:824] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:27.521834 3472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa: No bootstrap required, opened a new log
I20260924 23:06:27.523624 3472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e337447ef1e4c8887e11a685d27ecaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 23:06:27.523780 3472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:27.523828 3472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e337447ef1e4c8887e11a685d27ecaa, State: Initialized, Role: FOLLOWER
I20260924 23:06:27.524001 3472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e337447ef1e4c8887e11a685d27ecaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 23:06:27.524104 3472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:27.524138 3472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:27.524223 3472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:27.525040 3472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e337447ef1e4c8887e11a685d27ecaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 23:06:27.525197 3472 leader_election.cc:302] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e337447ef1e4c8887e11a685d27ecaa; no voters:
I20260924 23:06:27.525364 3472 leader_election.cc:288] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:27.525401 3477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:27.525519 3477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [term 1 LEADER]: Becoming Leader. State: Replica: 1e337447ef1e4c8887e11a685d27ecaa, State: Running, Role: LEADER
I20260924 23:06:27.525627 3472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:27.525646 3477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e337447ef1e4c8887e11a685d27ecaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } }
I20260924 23:06:27.525967 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e337447ef1e4c8887e11a685d27ecaa. Latest consensus state: current_term: 1 leader_uuid: "1e337447ef1e4c8887e11a685d27ecaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e337447ef1e4c8887e11a685d27ecaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } } }
I20260924 23:06:27.526103 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:27.526396 3484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:27.526906 3484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:27.525986 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e337447ef1e4c8887e11a685d27ecaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e337447ef1e4c8887e11a685d27ecaa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39405 } } }
I20260924 23:06:27.527956 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa [sys.catalog]: This master's current role is: LEADER
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:27.528692 3494 mvcc.cc:205] Tried to move back new op lower bound from 7333032704115740672 to 7333032704105304064. Current Snapshot: MvccSnapshot[applied={T|T < 7333032704115740672}]
I20260924 23:06:27.529466 3484 catalog_manager.cc:1347] Generated new cluster ID: 1f25daf7ac7043b69e31afcd500d85bb
I20260924 23:06:27.529513 3484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:27.532734 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:27.520699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.0.98.62:39405)
0924 23:06:27.520903 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:27.520913 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:27.520949 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:06:27.521577 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:27.521586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:27.521669 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:27.521826 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:27.522517 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.523489 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.524550 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.524968 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.525107 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:27.528828 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:27.528831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:27.528832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:27.528843 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:27.530808 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:27.531254 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:27.531257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:27.531257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:27.531349 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:27.531843 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:27.531844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:27.531844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:27.532104 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:27.532274 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:27.532382 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 23:06:27.532531 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260924 23:06:27.537288 3484 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:27.538028 3484 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:27.545369 3484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e337447ef1e4c8887e11a685d27ecaa: Generated new TSK 0
I20260924 23:06:27.545550 3484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:27.610855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39405
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:27.694226 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:27.694445 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:27.694476 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:27.695698 3498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:27.695748 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:27.695775 3498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:27.697283 3498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:27.697568 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:27.697824 3498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:27.703353 3503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:27.703320 3506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:27.703352 3504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:27.703675 3498 server_base.cc:1034] running on GCE node
I20260924 23:06:27.703931 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:27.704218 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:27.705374 3498 hybrid_clock.cc:648] HybridClock initialized: now 1790291187705348 us; error 43 us; skew 500 ppm
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:27.707934 3498 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:27.708385 3498 webserver.cc:460] Webserver started at http://127.0.98.1:41041/ using document root <none> and password file <none>
I20260924 23:06:27.708616 3498 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:27.708685 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:27.708839 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:27.709985 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "0553927868264fc4a4ac232221ea5190"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "0381f016203c3bd250dae53f05a90d57"
server_key_iv: "7418dd8f0cf6d9435c3fe5cc96a2fbf9"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.710354 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0553927868264fc4a4ac232221ea5190"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "0381f016203c3bd250dae53f05a90d57"
server_key_iv: "7418dd8f0cf6d9435c3fe5cc96a2fbf9"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.711661 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:27.712355 3513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.712539 3498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:27.712646 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "0553927868264fc4a4ac232221ea5190"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "0381f016203c3bd250dae53f05a90d57"
server_key_iv: "7418dd8f0cf6d9435c3fe5cc96a2fbf9"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.712730 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:27.728276 3498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:27.730285 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:27.730449 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:27.730683 3498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:27.731010 3498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:27.731063 3498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.731104 3498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:27.731133 3498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.737097 3498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38491
I20260924 23:06:27.737142 3626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38491 every 8 connection(s)
I20260924 23:06:27.737459 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:27.746215 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3498
I20260924 23:06:27.746343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:27.746449 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:27.738195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55683 (local address 127.0.98.62:39405)
0924 23:06:27.738239 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:27.738241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:27.738489 (+ 248us) server_negotiation.cc:407] Connection header received
0924 23:06:27.739020 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:27.739021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:27.739033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:27.739088 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:27.740028 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.740525 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.741510 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.741713 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.741797 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:27.743909 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:27.743910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:27.743911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:27.743917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:27.745070 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:27.745428 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:27.745429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:27.745429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:27.745477 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:27.745683 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:27.745683 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:27.745683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:27.745779 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:27.745829 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:27.746226 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 23:06:27.746347 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:27.746475 3629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:27.738189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39405 (local address 127.0.98.1:55683)
0924 23:06:27.738364 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:27.738401 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:27.738916 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:27.739170 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:27.739178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:27.739726 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:27.739930 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:27.739934 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.740617 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.740618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:27.741405 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:27.741407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.741807 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.741808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:27.741998 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:27.742466 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:27.742474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:27.743820 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:27.745143 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:27.745149 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:27.745150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:27.745363 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:27.745533 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:27.745534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:27.745534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:27.745630 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:27.745922 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:27.745942 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:27.746070 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 23:06:27.746168 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 23:06:27.746313 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260924 23:06:27.746553 392 external_mini_cluster.cc:1428] Setting key 29abda3c0a1611f87af0cf152f83277d
I20260924 23:06:27.746997 3627 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39405
I20260924 23:06:27.747088 3627 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:27.747318 3627 heartbeater.cc:511] Master 127.0.98.62:39405 requested a full tablet report, sending...
I20260924 23:06:27.747787 3436 ts_manager.cc:194] Registered new tserver with Master: 0553927868264fc4a4ac232221ea5190 (127.0.98.1:38491)
I20260924 23:06:27.748404 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55683
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:27.772704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39405
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:27.855477 3632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:27.855685 3632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:27.855707 3632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:27.856789 3632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:27.856825 3632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:27.856858 3632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:27.858333 3632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:27.858513 3632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:27.858700 3632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:27.863603 3640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:27.863633 3637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:27.863649 3638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:27.863757 3632 server_base.cc:1034] running on GCE node
I20260924 23:06:27.864094 3632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:27.864333 3632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:27.865492 3632 hybrid_clock.cc:648] HybridClock initialized: now 1790291187865483 us; error 31 us; skew 500 ppm
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:27.868105 3632 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:27.868489 3632 webserver.cc:460] Webserver started at http://127.0.98.2:39727/ using document root <none> and password file <none>
I20260924 23:06:27.868665 3632 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:27.868705 3632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:27.868794 3632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:27.869801 3632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "293db8d8e2f94f0082fe1bee487b630b"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "1100f5e6daa4b344a006183ca064ae89"
server_key_iv: "69caece7fc9212e9ca96c212977162d9"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.870193 3632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "293db8d8e2f94f0082fe1bee487b630b"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "1100f5e6daa4b344a006183ca064ae89"
server_key_iv: "69caece7fc9212e9ca96c212977162d9"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.871481 3632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:27.872145 3647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.872296 3632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:27.872395 3632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "293db8d8e2f94f0082fe1bee487b630b"
format_stamp: "Formatted at 2026-09-24 23:06:27 on dist-test-slave-ph9j"
server_key: "1100f5e6daa4b344a006183ca064ae89"
server_key_iv: "69caece7fc9212e9ca96c212977162d9"
server_key_version: "encryptionkey@0"
I20260924 23:06:27.872452 3632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:27.884822 3632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:27.886801 3632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:27.886909 3632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:27.887099 3632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:27.887457 3632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:27.887495 3632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.887517 3632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:27.887559 3632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:27.893158 3632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36203
I20260924 23:06:27.893231 3760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36203 every 8 connection(s)
I20260924 23:06:27.893558 3632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:27.897662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3632
I20260924 23:06:27.897784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:27.898037 392 external_mini_cluster.cc:1428] Setting key 3b2adfccf08e996e8a2c32168a4e84a3
Sep 24 23:06:27 dist-test-slave-ph9j krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:27.903921 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:27.894287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48033 (local address 127.0.98.62:39405)
0924 23:06:27.894331 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:27.894333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:27.894522 (+ 189us) server_negotiation.cc:407] Connection header received
0924 23:06:27.895052 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:27.895053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:27.895064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:27.895126 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:27.896069 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.896551 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.897604 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.897845 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.897965 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:27.900815 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:27.900817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:27.900817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:27.900824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:27.902412 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:27.902924 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:27.902925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:27.902925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:27.902975 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:27.903273 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:27.903274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:27.903274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:27.903375 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:27.903425 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:27.903784 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 23:06:27.903848 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:06:27.904261 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:27.894244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39405 (local address 127.0.98.2:48033)
0924 23:06:27.894397 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:27.894438 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:27.894951 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:27.895203 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:27.895206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:27.895759 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:27.895993 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:27.895998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.896636 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.896636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:27.897519 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:27.897522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:27.897974 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:27.897975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:27.898121 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:27.898718 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:27.898728 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:27.900697 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:27.902497 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:27.902504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:27.902506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:27.902838 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:27.903042 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:27.903043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:27.903043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:27.903195 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:27.903441 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:27.903448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:27.903632 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 23:06:27.903767 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 23:06:27.903953 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260924 23:06:27.904783 3761 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39405
I20260924 23:06:27.904897 3761 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:27.905115 3761 heartbeater.cc:511] Master 127.0.98.62:39405 requested a full tablet report, sending...
I20260924 23:06:27.905536 3436 ts_manager.cc:194] Registered new tserver with Master: 293db8d8e2f94f0082fe1bee487b630b (127.0.98.2:36203)
I20260924 23:06:27.905987 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48033
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:27.924032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39405
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:28.005725 3766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:28.006042 3766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:28.006196 3766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:28.007508 3766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:28.007561 3766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:28.007591 3766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:28.009181 3766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:28.009436 3766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:28.009685 3766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:28.014917 3774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:28.014925 3772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:28.014964 3766 server_base.cc:1034] running on GCE node
W20260924 23:06:28.014955 3771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:28.015338 3766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:28.015607 3766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:28.016776 3766 hybrid_clock.cc:648] HybridClock initialized: now 1790291188016758 us; error 34 us; skew 500 ppm
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:28.019325 3766 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:28.019788 3766 webserver.cc:460] Webserver started at http://127.0.98.3:33917/ using document root <none> and password file <none>
I20260924 23:06:28.019975 3766 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:28.020015 3766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:28.020104 3766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:28.021111 3766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "d6b735cf127f41e2bb9f3401b49d7b4a"
format_stamp: "Formatted at 2026-09-24 23:06:28 on dist-test-slave-ph9j"
server_key: "524e016dfcad85e35eb77e7bc028533a"
server_key_iv: "c73dfee230d8230d63d2a518a066d92b"
server_key_version: "encryptionkey@0"
I20260924 23:06:28.021406 3766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d6b735cf127f41e2bb9f3401b49d7b4a"
format_stamp: "Formatted at 2026-09-24 23:06:28 on dist-test-slave-ph9j"
server_key: "524e016dfcad85e35eb77e7bc028533a"
server_key_iv: "c73dfee230d8230d63d2a518a066d92b"
server_key_version: "encryptionkey@0"
I20260924 23:06:28.022814 3766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:28.023530 3781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:28.023736 3766 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:28.023837 3766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "d6b735cf127f41e2bb9f3401b49d7b4a"
format_stamp: "Formatted at 2026-09-24 23:06:28 on dist-test-slave-ph9j"
server_key: "524e016dfcad85e35eb77e7bc028533a"
server_key_iv: "c73dfee230d8230d63d2a518a066d92b"
server_key_version: "encryptionkey@0"
I20260924 23:06:28.023916 3766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:28.048998 3766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:28.050969 3766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:28.051128 3766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:28.051364 3766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:28.051687 3766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:28.051717 3766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:28.051741 3766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:28.051787 3766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:28.057655 3766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44227
I20260924 23:06:28.057708 3894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44227 every 8 connection(s)
I20260924 23:06:28.058089 3766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:28.058923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3766
I20260924 23:06:28.059023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:28.059214 392 external_mini_cluster.cc:1428] Setting key 78642b47d687afc9749d5451ea027910
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:28.067158 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:28.058845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55073 (local address 127.0.98.62:39405)
0924 23:06:28.058918 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:28.058921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:28.059099 (+ 178us) server_negotiation.cc:407] Connection header received
0924 23:06:28.059672 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:28.059673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:28.059684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:28.059728 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:28.060769 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:28.061269 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:28.062344 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:28.062559 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:28.062648 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:28.064609 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:28.064610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:28.064611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:28.064619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:28.065789 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:28.066214 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:28.066215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:28.066215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:28.066278 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:28.066533 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:28.066533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:28.066534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:28.066646 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:28.066716 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:28.066983 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 23:06:28.067068 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:06:28.067349 3897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:28.058800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39405 (local address 127.0.98.3:55073)
0924 23:06:28.058950 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:28.058999 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:28.059585 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:28.059798 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:28.059802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:28.060485 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:28.060673 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:28.060677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:28.061379 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:28.061381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:28.062258 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:28.062260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:28.062658 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:28.062659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:28.062778 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:28.063207 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:28.063214 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:28.064514 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:28.065908 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:28.065914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:28.065916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:28.066134 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:28.066346 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:28.066347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:28.066348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:28.066452 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:28.066719 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:28.066726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:28.066856 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 23:06:28.066970 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:06:28.067121 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260924 23:06:28.067806 3895 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39405
I20260924 23:06:28.067893 3895 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:28.068020 3895 heartbeater.cc:511] Master 127.0.98.62:39405 requested a full tablet report, sending...
I20260924 23:06:28.068360 3436 ts_manager.cc:194] Registered new tserver with Master: d6b735cf127f41e2bb9f3401b49d7b4a (127.0.98.3:44227)
I20260924 23:06:28.068679 3436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55073
I20260924 23:06:28.070891 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291188, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291188, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:28.096911 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:28.086100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.0.98.62:39405)
0924 23:06:28.086175 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:28.086177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:28.086282 (+ 105us) server_negotiation.cc:407] Connection header received
0924 23:06:28.086370 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:28.086371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:28.086381 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:28.086435 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:28.087053 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:28.087547 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:28.088306 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:28.088504 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:28.088582 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:28.094784 (+ 6202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:28.094786 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:28.094786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:28.094793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:28.095951 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:28.096261 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:28.096262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:28.096262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:28.096311 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:28.096483 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:28.096484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:28.096484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:28.096626 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:28.096707 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:28.096788 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 23:06:28.096845 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 23:06:28.097206 3436 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44174
I20260924 23:06:28.097707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3498
I20260924 23:06:28.102101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3632
I20260924 23:06:28.106384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3766
I20260924 23:06:28.110608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3404
2026-09-24T23:06:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1875 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3907](info): set up 2 sockets
Sep 24 23:06:28 dist-test-slave-ph9j krb5kdc[3907](info): commencing operation
krb5kdc: starting...
W20260924 23:06:29.149709 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:29.269712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:29.349298 3920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:29.349501 3920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:29.349524 3920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:29.350638 3920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:29.350675 3920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:29.350689 3920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:29.350703 3920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:29.350713 3920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:29.352079 3920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:29.352264 3920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:29.352432 3920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:29.357372 3925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:29.357436 3928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:29.357506 3926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:29.357590 3920 server_base.cc:1034] running on GCE node
I20260924 23:06:29.357944 3920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:29.358291 3920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:29.359455 3920 hybrid_clock.cc:648] HybridClock initialized: now 1790291189359443 us; error 49 us; skew 500 ppm
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:29.362013 3920 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:29.362452 3920 webserver.cc:460] Webserver started at http://127.0.98.62:42435/ using document root <none> and password file <none>
I20260924 23:06:29.362629 3920 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:29.362668 3920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:29.362751 3920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:29.363721 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "57bea5ac8e4f4689b76612938b6b302a"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "2f2898415dec6f45c0420247c6b9700e"
server_key_iv: "d954ba75673d12572d00fa7b5dd12a1e"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.364023 3920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "57bea5ac8e4f4689b76612938b6b302a"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "2f2898415dec6f45c0420247c6b9700e"
server_key_iv: "d954ba75673d12572d00fa7b5dd12a1e"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.365298 3920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:29.366110 3935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.366285 3920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:29.366366 3920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "57bea5ac8e4f4689b76612938b6b302a"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "2f2898415dec6f45c0420247c6b9700e"
server_key_iv: "d954ba75673d12572d00fa7b5dd12a1e"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.366425 3920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:29.392339 3920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:29.394850 3920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:29.394982 3920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:29.399000 3920 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41151
I20260924 23:06:29.399063 3987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41151 every 8 connection(s)
I20260924 23:06:29.399402 3920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:29.400274 3988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:29.402668 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a: Bootstrap starting.
I20260924 23:06:29.403290 3988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:29.403604 3988 log.cc:824] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:29.404340 3988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a: No bootstrap required, opened a new log
I20260924 23:06:29.404855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 3920
I20260924 23:06:29.404959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:29.405187 392 external_mini_cluster.cc:1428] Setting key 0502b26b77c6456fea68286dec935a24
I20260924 23:06:29.406798 3988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bea5ac8e4f4689b76612938b6b302a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41151 } }
I20260924 23:06:29.407038 3988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:29.407080 3988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea5ac8e4f4689b76612938b6b302a, State: Initialized, Role: FOLLOWER
I20260924 23:06:29.407220 3988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bea5ac8e4f4689b76612938b6b302a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41151 } }
I20260924 23:06:29.407325 3988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:29.407370 3988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:29.407423 3988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:29.408557 3988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bea5ac8e4f4689b76612938b6b302a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41151 } }
I20260924 23:06:29.408669 3988 leader_election.cc:302] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57bea5ac8e4f4689b76612938b6b302a; no voters:
I20260924 23:06:29.408818 3988 leader_election.cc:288] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:29.408893 3993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:29.409067 3993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [term 1 LEADER]: Becoming Leader. State: Replica: 57bea5ac8e4f4689b76612938b6b302a, State: Running, Role: LEADER
I20260924 23:06:29.409071 3988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:29.409837 3993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bea5ac8e4f4689b76612938b6b302a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41151 } }
I20260924 23:06:29.410511 3994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57bea5ac8e4f4689b76612938b6b302a. Latest consensus state: current_term: 1 leader_uuid: "57bea5ac8e4f4689b76612938b6b302a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bea5ac8e4f4689b76612938b6b302a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41151 } } }
I20260924 23:06:29.410717 3994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [sys.catalog]: This master's current role is: LEADER
W20260924 23:06:29.410986 4007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 23:06:29.411094 3995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57bea5ac8e4f4689b76612938b6b302a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57bea5ac8e4f4689b76612938b6b302a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41151 } } }
W20260924 23:06:29.411131 4007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:06:29.411204 3995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:29.411326 4009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:29.411546 4009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:29.413205 4009 catalog_manager.cc:1347] Generated new cluster ID: f433c45747194d53af74bc6da35c3d00
I20260924 23:06:29.413259 4009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:29.417745 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.405761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.98.62:41151)
0924 23:06:29.405996 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:29.406007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:29.406045 (+ 38us) server_negotiation.cc:407] Connection header received
0924 23:06:29.406739 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:29.406749 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:29.406823 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:29.407006 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:29.407960 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.409443 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.410683 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.411321 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.411464 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.414192 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:29.414196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:29.414196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:29.414208 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:29.415931 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.416303 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.416306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.416307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.416718 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.416948 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.416949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.416949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.417188 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:29.417370 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:29.417449 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 23:06:29.417599 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":125,"threads_started":1}
I20260924 23:06:29.438495 4009 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:29.439270 4009 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:29.443097 4009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57bea5ac8e4f4689b76612938b6b302a: Generated new TSK 0
I20260924 23:06:29.443334 4009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:29.473172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41151
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:29.553833 4014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:29.554126 4014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:29.554152 4014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:29.555358 4014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:29.555399 4014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:29.555426 4014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:29.556938 4014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:29.557180 4014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:29.557462 4014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:29.562755 4022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:29.562747 4019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:29.562742 4020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:29.562963 4014 server_base.cc:1034] running on GCE node
I20260924 23:06:29.563253 4014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:29.563512 4014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:29.564679 4014 hybrid_clock.cc:648] HybridClock initialized: now 1790291189564680 us; error 43 us; skew 500 ppm
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:29.567286 4014 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:29.567745 4014 webserver.cc:460] Webserver started at http://127.0.98.1:40147/ using document root <none> and password file <none>
I20260924 23:06:29.567934 4014 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:29.567973 4014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:29.568060 4014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:29.569080 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "e1a623db56384601bd839418676b48cc"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "517ab04cc641ad8f5d9036fcb0cbc037"
server_key_iv: "1478cfc296b2f083d9c554396b2921a1"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.569393 4014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e1a623db56384601bd839418676b48cc"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "517ab04cc641ad8f5d9036fcb0cbc037"
server_key_iv: "1478cfc296b2f083d9c554396b2921a1"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.570739 4014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:29.571487 4029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.571671 4014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:29.571759 4014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "e1a623db56384601bd839418676b48cc"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "517ab04cc641ad8f5d9036fcb0cbc037"
server_key_iv: "1478cfc296b2f083d9c554396b2921a1"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.571834 4014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:29.587437 4014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:29.589859 4014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:29.590080 4014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:29.590422 4014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:29.590745 4014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:29.590803 4014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.590840 4014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:29.590893 4014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.596789 4014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38867
I20260924 23:06:29.596830 4142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38867 every 8 connection(s)
I20260924 23:06:29.597172 4014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:29.598084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4014
I20260924 23:06:29.598208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:29.598397 392 external_mini_cluster.cc:1428] Setting key 7b509a66ec6b87a577ba1cd69ae1ea1d
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 23:06:29.607197 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.597871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35857 (local address 127.0.98.62:41151)
0924 23:06:29.597957 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:29.597959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:29.598199 (+ 240us) server_negotiation.cc:407] Connection header received
0924 23:06:29.599236 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:29.599237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:29.599255 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:29.599328 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:29.599957 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.600609 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.601638 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.601863 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.601969 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.604378 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:29.604379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:29.604380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:29.604387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:29.605791 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.606225 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.606227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.606227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.606286 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.606619 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.606620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.606620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.606723 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:29.606783 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:29.607057 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 23:06:29.607129 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 23:06:29.607359 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.597857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41151 (local address 127.0.98.1:35857)
0924 23:06:29.598070 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:29.598114 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:29.599135 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:29.599383 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:29.599386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:29.599670 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:29.599865 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:29.599869 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.600708 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.600709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:29.601530 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:29.601532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.601975 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.601976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:29.602111 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.602670 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:29.602678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:29.604280 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:29.605900 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:29.605907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:29.605909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:29.606135 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:29.606388 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:29.606390 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:29.606390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:29.606542 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:29.606807 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:29.606812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:29.606934 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 23:06:29.607029 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 23:06:29.607153 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":142,"threads_started":1}
I20260924 23:06:29.607880 4143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41151
I20260924 23:06:29.607990 4143 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:29.608186 4143 heartbeater.cc:511] Master 127.0.98.62:41151 requested a full tablet report, sending...
I20260924 23:06:29.608726 3952 ts_manager.cc:194] Registered new tserver with Master: e1a623db56384601bd839418676b48cc (127.0.98.1:38867)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 23:06:29.609534 3952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:29.625105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41151
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:29.705901 4148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:29.706102 4148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:29.706125 4148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:29.707285 4148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:29.707330 4148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:29.707396 4148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:29.708873 4148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:29.709124 4148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:29.709419 4148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:29.714501 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:29.714581 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:29.714604 4148 server_base.cc:1034] running on GCE node
W20260924 23:06:29.714501 4153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:29.714908 4148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:29.715114 4148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:29.716248 4148 hybrid_clock.cc:648] HybridClock initialized: now 1790291189716227 us; error 37 us; skew 500 ppm
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:29.718832 4148 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:29.719226 4148 webserver.cc:460] Webserver started at http://127.0.98.2:37203/ using document root <none> and password file <none>
I20260924 23:06:29.719408 4148 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:29.719449 4148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:29.719539 4148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:29.720520 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "87156b8fc2d243fbb505ef9b4970edf6"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "b2dfff42461297f9479b4cad9a934be9"
server_key_iv: "17390e39c20337a59cda691583dbd1c4"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.720824 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "87156b8fc2d243fbb505ef9b4970edf6"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "b2dfff42461297f9479b4cad9a934be9"
server_key_iv: "17390e39c20337a59cda691583dbd1c4"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.722114 4148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:29.722762 4163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.722942 4148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:29.723062 4148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "87156b8fc2d243fbb505ef9b4970edf6"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "b2dfff42461297f9479b4cad9a934be9"
server_key_iv: "17390e39c20337a59cda691583dbd1c4"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.723136 4148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:29.742000 4148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:29.744380 4148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:29.744550 4148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:29.744796 4148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:29.745115 4148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:29.745169 4148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.745211 4148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:29.745239 4148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.751080 4148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40565
I20260924 23:06:29.751117 4276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40565 every 8 connection(s)
I20260924 23:06:29.751457 4148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:29.759776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4148
I20260924 23:06:29.759910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:29.760145 392 external_mini_cluster.cc:1428] Setting key 98f5d5686c38bdd36db16687b0b961c3
I20260924 23:06:29.760969 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.752187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58445 (local address 127.0.98.62:41151)
0924 23:06:29.752258 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:29.752260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:29.752469 (+ 209us) server_negotiation.cc:407] Connection header received
0924 23:06:29.753395 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:29.753396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:29.753405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:29.753457 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:29.754131 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.754620 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.755561 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.755742 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.755793 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.757789 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:29.757790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:29.757791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:29.757797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:29.759257 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.759695 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.759696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.759697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.759757 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.760056 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.760058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.760058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.760215 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:29.760278 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:29.760708 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 23:06:29.760825 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:29.761059 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.752165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41151 (local address 127.0.98.2:58445)
0924 23:06:29.752349 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:29.752384 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:29.753296 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:29.753518 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:29.753521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:29.753767 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:29.754055 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:29.754061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.754697 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.754697 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:29.755489 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:29.755491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.755812 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.755812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:29.755908 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.756357 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:29.756365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:29.757693 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:29.759360 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:29.759366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:29.759368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:29.759599 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:29.759834 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:29.759835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:29.759835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:29.759994 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:29.760308 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:29.760313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:29.760498 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 23:06:29.760656 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 23:06:29.760826 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260924 23:06:29.761636 4277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41151
I20260924 23:06:29.761756 4277 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:29.761988 4277 heartbeater.cc:511] Master 127.0.98.62:41151 requested a full tablet report, sending...
I20260924 23:06:29.762373 3952 ts_manager.cc:194] Registered new tserver with Master: 87156b8fc2d243fbb505ef9b4970edf6 (127.0.98.2:40565)
I20260924 23:06:29.762759 3952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58445
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:29.786092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41151
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:29.870226 4282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:29.870445 4282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:29.870469 4282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:29.871631 4282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:29.871678 4282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:29.871706 4282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:29.873247 4282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:29.873499 4282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:29.873730 4282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:29.878957 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:29.878928 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:29.879068 4282 server_base.cc:1034] running on GCE node
W20260924 23:06:29.878942 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:29.879385 4282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:29.879601 4282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:29.880744 4282 hybrid_clock.cc:648] HybridClock initialized: now 1790291189880727 us; error 33 us; skew 500 ppm
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:29.883435 4282 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:29.883898 4282 webserver.cc:460] Webserver started at http://127.0.98.3:36679/ using document root <none> and password file <none>
I20260924 23:06:29.884132 4282 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:29.884178 4282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:29.884348 4282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:29.885406 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "f3b8291a32c149f7abe570af96a59835"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "9d91f01e218e3ef96a2f80a4ba9b606e"
server_key_iv: "95c4c9c9e049c444b1e3b47102b2cdab"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.885778 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f3b8291a32c149f7abe570af96a59835"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "9d91f01e218e3ef96a2f80a4ba9b606e"
server_key_iv: "95c4c9c9e049c444b1e3b47102b2cdab"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.887156 4282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:29.887910 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.888098 4282 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:29.888173 4282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "f3b8291a32c149f7abe570af96a59835"
format_stamp: "Formatted at 2026-09-24 23:06:29 on dist-test-slave-ph9j"
server_key: "9d91f01e218e3ef96a2f80a4ba9b606e"
server_key_iv: "95c4c9c9e049c444b1e3b47102b2cdab"
server_key_version: "encryptionkey@0"
I20260924 23:06:29.888268 4282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:29.900437 4282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:29.902379 4282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:29.902549 4282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:29.902781 4282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:29.903088 4282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:29.903121 4282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.903144 4282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:29.903193 4282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:29.909003 4282 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35353
I20260924 23:06:29.909075 4410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35353 every 8 connection(s)
I20260924 23:06:29.909448 4282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:29.911005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4282
I20260924 23:06:29.911163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:29.911360 392 external_mini_cluster.cc:1428] Setting key b7bbda340ba414d34005aa8e90b14a44
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:29.918841 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.910279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59835 (local address 127.0.98.62:41151)
0924 23:06:29.910321 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:29.910323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:29.910569 (+ 246us) server_negotiation.cc:407] Connection header received
0924 23:06:29.911764 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:29.911765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:29.911777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:29.911835 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:29.912557 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.913056 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.914065 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.914242 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.914294 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.916354 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:29.916355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:29.916355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:29.916362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:29.917624 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.918095 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.918096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.918096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.918147 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.918353 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.918354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.918354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.918451 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:29.918509 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:29.918720 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 23:06:29.918779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:06:29.919040 4413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.910284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41151 (local address 127.0.98.3:59835)
0924 23:06:29.910428 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:29.910464 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:29.911657 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:29.911917 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:29.911921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:29.912285 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:29.912480 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:29.912484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.913133 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.913134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:29.913993 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:29.913995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.914308 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.914309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:29.914406 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.914941 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:29.914949 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:29.916268 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:29.917712 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:29.917719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:29.917721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:29.918032 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:29.918206 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:29.918207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:29.918207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:29.918301 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:29.918504 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:29.918508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:29.918615 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 23:06:29.918692 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 23:06:29.918803 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260924 23:06:29.919634 4411 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41151
I20260924 23:06:29.919764 4411 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:29.919955 4411 heartbeater.cc:511] Master 127.0.98.62:41151 requested a full tablet report, sending...
I20260924 23:06:29.920365 3952 ts_manager.cc:194] Registered new tserver with Master: f3b8291a32c149f7abe570af96a59835 (127.0.98.3:35353)
I20260924 23:06:29.920749 3952 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59835
I20260924 23:06:29.922993 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:29.945484 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.937829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.0.98.1:38867)
0924 23:06:29.938061 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:29.938068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:29.938085 (+ 17us) server_negotiation.cc:407] Connection header received
0924 23:06:29.938112 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:29.938114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:29.938128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:29.938202 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:29.938781 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.939538 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.940248 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.940568 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.940619 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.942890 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:29.942901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:29.942901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:29.942910 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:29.944490 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.944812 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.944815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.944815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.944868 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.945048 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.945049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.945049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.945174 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:29.945273 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:29.945337 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 23:06:29.945397 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":164,"threads_started":1}
Sep 24 23:06:29 dist-test-slave-ph9j krb5kdc[3907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:29.953568 3992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:29.946748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.0.98.62:41151)
0924 23:06:29.946814 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:29.946816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:29.946862 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:06:29.946926 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:29.946927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:29.946937 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:29.946981 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:29.947430 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.948091 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.949040 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:29.949246 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:29.949334 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:29.951461 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:29.951462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:29.951462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:29.951469 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:29.952614 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.952932 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.952933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.952933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.952980 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:29.953138 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:29.953138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:29.953139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:29.953273 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:29.953341 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:29.953423 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 23:06:29.953478 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 23:06:29.953825 3952 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58266
I20260924 23:06:29.954236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4014
I20260924 23:06:29.959023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4148
I20260924 23:06:29.963531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4282
I20260924 23:06:29.967629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 3920
2026-09-24T23:06:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1857 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:30 dist-test-slave-ph9j krb5kdc[4423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:30 dist-test-slave-ph9j krb5kdc[4423](info): set up 2 sockets
Sep 24 23:06:30 dist-test-slave-ph9j krb5kdc[4423](info): commencing operation
krb5kdc: starting...
W20260924 23:06:31.009841 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:31.128990 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40457
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:31.209132 4436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:31.209342 4436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:31.209367 4436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:31.210510 4436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:31.210551 4436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:31.210564 4436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:31.210577 4436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:31.210589 4436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:31.212020 4436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:31.212205 4436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:31.212380 4436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:31.217231 4444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:31.217267 4442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:31.217268 4441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:31.217320 4436 server_base.cc:1034] running on GCE node
I20260924 23:06:31.217680 4436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:31.218036 4436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:31.219211 4436 hybrid_clock.cc:648] HybridClock initialized: now 1790291191219199 us; error 46 us; skew 500 ppm
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:31.221604 4436 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:31.222013 4436 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:31.222087 4436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:31.222246 4436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:31.223340 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "28c79734260f4be5a7e394fb44a4dd3d"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "0e92fc3fb94ff072ba4fc5715a9e476a"
server_key_iv: "d30283831728620f71f9eb176fa4543b"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.223716 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "28c79734260f4be5a7e394fb44a4dd3d"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "0e92fc3fb94ff072ba4fc5715a9e476a"
server_key_iv: "d30283831728620f71f9eb176fa4543b"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.225057 4436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:31.225931 4450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.226133 4436 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:31.226233 4436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "28c79734260f4be5a7e394fb44a4dd3d"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "0e92fc3fb94ff072ba4fc5715a9e476a"
server_key_iv: "d30283831728620f71f9eb176fa4543b"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.226320 4436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:31.235303 4436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:31.237119 4436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:31.237273 4436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:31.241310 4436 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40457
I20260924 23:06:31.241338 4502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40457 every 8 connection(s)
I20260924 23:06:31.241633 4436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:31.242621 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:31.243508 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4436
I20260924 23:06:31.243613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:31.243790 392 external_mini_cluster.cc:1428] Setting key 24b8d6159365da589065ef5b70b46d40
I20260924 23:06:31.245779 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: Bootstrap starting.
I20260924 23:06:31.246413 4503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:31.246693 4503 log.cc:824] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:31.247344 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: No bootstrap required, opened a new log
I20260924 23:06:31.249033 4503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c79734260f4be5a7e394fb44a4dd3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40457 } }
I20260924 23:06:31.249186 4503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:31.249233 4503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c79734260f4be5a7e394fb44a4dd3d, State: Initialized, Role: FOLLOWER
I20260924 23:06:31.249370 4503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c79734260f4be5a7e394fb44a4dd3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40457 } }
I20260924 23:06:31.249466 4503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:31.249509 4503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:31.249557 4503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:31.250187 4503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c79734260f4be5a7e394fb44a4dd3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40457 } }
I20260924 23:06:31.250304 4503 leader_election.cc:302] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28c79734260f4be5a7e394fb44a4dd3d; no voters:
I20260924 23:06:31.250460 4503 leader_election.cc:288] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:31.250538 4508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:31.250676 4508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [term 1 LEADER]: Becoming Leader. State: Replica: 28c79734260f4be5a7e394fb44a4dd3d, State: Running, Role: LEADER
I20260924 23:06:31.250703 4503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:31.250768 4508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c79734260f4be5a7e394fb44a4dd3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40457 } }
I20260924 23:06:31.251045 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28c79734260f4be5a7e394fb44a4dd3d. Latest consensus state: current_term: 1 leader_uuid: "28c79734260f4be5a7e394fb44a4dd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c79734260f4be5a7e394fb44a4dd3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40457 } } }
I20260924 23:06:31.251133 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:31.251055 4510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28c79734260f4be5a7e394fb44a4dd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28c79734260f4be5a7e394fb44a4dd3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40457 } } }
I20260924 23:06:31.251271 4510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d [sys.catalog]: This master's current role is: LEADER
W20260924 23:06:31.252519 4523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:06:31.252575 4523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:06:31.252624 4514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:31.252807 4514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:31.254236 4514 catalog_manager.cc:1347] Generated new cluster ID: d32a1c857db547e98b1ef9b47219ce3f
I20260924 23:06:31.254297 4514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:31.254997 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.244308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.0.98.62:40457)
0924 23:06:31.244496 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.244506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.244537 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:06:31.245028 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.245036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.245103 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.245255 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:31.245988 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.247048 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.247876 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.248311 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.248432 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.251177 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:31.251183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:31.251183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:31.251195 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:31.253348 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.253741 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.253744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.253744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.253858 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.254105 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.254106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.254107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.254429 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:31.254582 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.254688 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.254835 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260924 23:06:31.264665 4514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:31.265192 4514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:31.267807 4514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: Generated new TSK 0
I20260924 23:06:31.267977 4514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:31.301003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40457
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:31.384202 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:31.384438 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:31.384464 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:31.385712 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:31.385758 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:31.385787 4529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:31.387372 4529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:31.387676 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:31.387949 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:31.393546 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:31.393539 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:31.393564 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:31.393904 4529 server_base.cc:1034] running on GCE node
I20260924 23:06:31.394192 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:31.394433 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:31.395586 4529 hybrid_clock.cc:648] HybridClock initialized: now 1790291191395569 us; error 38 us; skew 500 ppm
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:31.398394 4529 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:31.398792 4529 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:31.398864 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:31.399063 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:31.400292 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "709e13e1a7134235899dae64e69500e6"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "dd70a3f4399edcc5556c62ddab82294c"
server_key_iv: "5190247a523540ea054049331919b742"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.400656 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "709e13e1a7134235899dae64e69500e6"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "dd70a3f4399edcc5556c62ddab82294c"
server_key_iv: "5190247a523540ea054049331919b742"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.402259 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:31.403028 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.403187 4529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:31.403293 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "709e13e1a7134235899dae64e69500e6"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "dd70a3f4399edcc5556c62ddab82294c"
server_key_iv: "5190247a523540ea054049331919b742"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.403368 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:31.418710 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:31.420755 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:31.420938 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:31.421186 4529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:31.421511 4529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:31.421566 4529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.421608 4529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:31.421636 4529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.427332 4529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36095
I20260924 23:06:31.427402 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36095 every 8 connection(s)
I20260924 23:06:31.427677 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:31.436197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4529
I20260924 23:06:31.436290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:31.436507 392 external_mini_cluster.cc:1428] Setting key f75a89de13b4f6ef7f4648f781a80366
I20260924 23:06:31.437103 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.428416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52103 (local address 127.0.98.62:40457)
0924 23:06:31.428460 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.428462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.428670 (+ 208us) server_negotiation.cc:407] Connection header received
0924 23:06:31.429281 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.429282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.429295 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.429344 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:31.430425 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.430947 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.431948 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.432158 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.432246 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.434358 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:31.434360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:31.434360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:31.434368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:31.435588 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.436015 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.436016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.436017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.436087 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.436400 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.436401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.436401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.436520 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:31.436598 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.436928 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.437014 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:31.437331 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.428359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40457 (local address 127.0.98.1:52103)
0924 23:06:31.428541 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.428586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.429176 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.429432 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.429436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.429711 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:31.430324 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.430329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.431045 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.431046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.431850 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.431852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.432255 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.432256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.432392 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.432876 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:31.432884 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:31.434247 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:31.435686 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:31.435693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:31.435696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:31.435934 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:31.436162 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:31.436163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:31.436163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:31.436309 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:31.436599 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:31.436606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:31.436774 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.436903 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.437058 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260924 23:06:31.437943 4657 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40457
I20260924 23:06:31.438069 4657 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:31.438223 4657 heartbeater.cc:511] Master 127.0.98.62:40457 requested a full tablet report, sending...
I20260924 23:06:31.438802 4467 ts_manager.cc:194] Registered new tserver with Master: 709e13e1a7134235899dae64e69500e6 (127.0.98.1:36095)
I20260924 23:06:31.439432 4467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52103
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:31.464443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40457
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:31.545925 4662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:31.546116 4662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:31.546139 4662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:31.547298 4662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:31.547340 4662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:31.547367 4662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:31.548957 4662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:31.549190 4662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:31.549417 4662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:31.554416 4667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:31.554442 4668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:31.554441 4662 server_base.cc:1034] running on GCE node
W20260924 23:06:31.554535 4670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:31.554926 4662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:31.555171 4662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:31.556332 4662 hybrid_clock.cc:648] HybridClock initialized: now 1790291191556307 us; error 58 us; skew 500 ppm
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:31.558836 4662 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:31.559182 4662 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:31.559253 4662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:31.559403 4662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:31.560412 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e02e7510a364797ba4a1ba33fb32fdd"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "90dd0e74e2ed11b8ac64ccce730044e5"
server_key_iv: "02cd572dd48d2b8355a8b1b15276d268"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.560760 4662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e02e7510a364797ba4a1ba33fb32fdd"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "90dd0e74e2ed11b8ac64ccce730044e5"
server_key_iv: "02cd572dd48d2b8355a8b1b15276d268"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.562111 4662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:31.562889 4676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.563094 4662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:06:31.563198 4662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "3e02e7510a364797ba4a1ba33fb32fdd"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "90dd0e74e2ed11b8ac64ccce730044e5"
server_key_iv: "02cd572dd48d2b8355a8b1b15276d268"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.563268 4662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:31.575510 4662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:31.577589 4662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:31.577754 4662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:31.578029 4662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:31.578333 4662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:31.578389 4662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.578423 4662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:31.578473 4662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.584003 4662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36953
I20260924 23:06:31.584036 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36953 every 8 connection(s)
I20260924 23:06:31.584304 4662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:31.589289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4662
I20260924 23:06:31.589375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:31.589622 392 external_mini_cluster.cc:1428] Setting key baf7245ec8c73b92864ee6e4592a6ecf
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:31.594779 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.585139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52845 (local address 127.0.98.62:40457)
0924 23:06:31.585200 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.585203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.585407 (+ 204us) server_negotiation.cc:407] Connection header received
0924 23:06:31.586024 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.586026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.586038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.586094 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:31.587350 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.587907 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.588949 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.589156 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.589245 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.591936 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:31.591938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:31.591938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:31.591945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:31.593284 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.593791 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.593792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.593792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.593842 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.594154 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.594155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.594155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.594255 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:31.594306 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.594652 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.594717 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:06:31.595112 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.585125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40457 (local address 127.0.98.2:52845)
0924 23:06:31.585280 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.585316 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.585814 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.586170 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.586176 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.586739 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:31.587243 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.587248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.588014 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.588015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.588825 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.588826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.589253 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.589254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.589611 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.590301 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:31.590313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:31.591817 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:31.593365 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:31.593371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:31.593374 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:31.593705 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:31.593937 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:31.593938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:31.593938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:31.594080 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:31.594324 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:31.594330 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:31.594505 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.594635 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.594818 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260924 23:06:31.595676 4790 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40457
I20260924 23:06:31.595791 4790 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:31.595989 4790 heartbeater.cc:511] Master 127.0.98.62:40457 requested a full tablet report, sending...
I20260924 23:06:31.596467 4467 ts_manager.cc:194] Registered new tserver with Master: 3e02e7510a364797ba4a1ba33fb32fdd (127.0.98.2:36953)
I20260924 23:06:31.597009 4467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52845
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:31.616858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40457
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:31.696688 4795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:31.696938 4795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:31.696982 4795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:31.698158 4795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:31.698227 4795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:31.698273 4795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:31.699761 4795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:31.700029 4795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:31.700250 4795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:31.705161 4800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:31.705181 4803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:31.705214 4795 server_base.cc:1034] running on GCE node
W20260924 23:06:31.705164 4801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:31.705596 4795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:31.705785 4795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:31.706945 4795 hybrid_clock.cc:648] HybridClock initialized: now 1790291191706954 us; error 46 us; skew 500 ppm
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:31.709292 4795 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:31.709672 4795 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:31.709745 4795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:31.709957 4795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:31.710999 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "8cde94af7788482abb26297a2820a2ce"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "408b32e63c0e891ae69492d2b91742ae"
server_key_iv: "c3f8e92fb6f3dd50b008c20741122994"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.711375 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8cde94af7788482abb26297a2820a2ce"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "408b32e63c0e891ae69492d2b91742ae"
server_key_iv: "c3f8e92fb6f3dd50b008c20741122994"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.712667 4795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:31.713371 4809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.713541 4795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:31.713645 4795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "8cde94af7788482abb26297a2820a2ce"
format_stamp: "Formatted at 2026-09-24 23:06:31 on dist-test-slave-ph9j"
server_key: "408b32e63c0e891ae69492d2b91742ae"
server_key_iv: "c3f8e92fb6f3dd50b008c20741122994"
server_key_version: "encryptionkey@0"
I20260924 23:06:31.713719 4795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:31.730015 4795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:31.731853 4795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:31.732016 4795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:31.732247 4795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:31.732571 4795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:31.732626 4795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.732666 4795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:31.732694 4795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:31.738651 4795 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40595
I20260924 23:06:31.738686 4922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40595 every 8 connection(s)
I20260924 23:06:31.738981 4795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:31.741585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4795
I20260924 23:06:31.741703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:31.741981 392 external_mini_cluster.cc:1428] Setting key 6aa118cc1624a330ccbeb8f8933d6884
Sep 24 23:06:31 dist-test-slave-ph9j krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:31.747912 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.739751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33509 (local address 127.0.98.62:40457)
0924 23:06:31.739819 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.739822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.740013 (+ 191us) server_negotiation.cc:407] Connection header received
0924 23:06:31.740470 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.740471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.740480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.740530 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:31.741394 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.742080 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.743346 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.743531 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.743582 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.745506 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:31.745507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:31.745508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:31.745514 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:31.746760 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.747135 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.747136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.747136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.747193 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.747428 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.747429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.747429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.747523 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:31.747571 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.747790 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.747847 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:06:31.748152 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.739735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40457 (local address 127.0.98.3:33509)
0924 23:06:31.739892 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.739928 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.740374 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.740586 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.740590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.740838 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:31.741304 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.741308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.742267 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.742269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.743275 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.743278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.743599 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.743600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.743690 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.744100 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:31.744108 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:31.745427 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:31.746826 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:31.746831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:31.746833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:31.747069 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:31.747254 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:31.747255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:31.747256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:31.747370 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:31.747576 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:31.747580 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:31.747685 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.747763 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.747875 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260924 23:06:31.748658 4923 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40457
I20260924 23:06:31.748735 4923 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:31.748843 4923 heartbeater.cc:511] Master 127.0.98.62:40457 requested a full tablet report, sending...
I20260924 23:06:31.749145 4467 ts_manager.cc:194] Registered new tserver with Master: 8cde94af7788482abb26297a2820a2ce (127.0.98.3:40595)
I20260924 23:06:31.749526 4467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33509
I20260924 23:06:31.753660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:06:31.760986 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.755121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.0.98.62:40457)
0924 23:06:31.755182 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.755185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.755274 (+ 89us) server_negotiation.cc:407] Connection header received
0924 23:06:31.755365 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.755366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.755376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.755447 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:31.756013 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.756478 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.757174 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.757368 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.757455 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.758626 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:31.758627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:31.758627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:31.758634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:31.759810 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.760159 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.760160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.760160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.760219 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:31.760422 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:31.760423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:31.760424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:31.760566 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:31.760665 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.760784 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.760862 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:06:31.762653 4467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:31.764171 4467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:31.770398 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.766332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40595 (local address 127.0.0.1:39358)
0924 23:06:31.766600 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.766640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.766710 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.767069 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.767072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.767074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:31.767137 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.767139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.768029 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.768030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.769167 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.769169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.770095 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.770096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.770173 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.770187 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.770257 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.770303 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":36,"thread_start_us":63,"threads_started":1}
I20260924 23:06:31.770592 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.766347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36095 (local address 127.0.0.1:56208)
0924 23:06:31.766805 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.766812 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.766874 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.767122 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.767123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.767124 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:31.767153 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.767155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.768001 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.768002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.769315 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.769317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.770203 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.770204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.770265 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.770270 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.770329 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.770378 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":38,"thread_start_us":58,"threads_started":1}
I20260924 23:06:31.770655 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.766509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.98.3:40595)
0924 23:06:31.766782 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.766786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.766795 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:31.766838 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.766839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.766851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.766968 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:31.767221 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.767909 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.769255 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.770010 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.770087 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.770147 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.770517 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.770567 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":88,"threads_started":1}
I20260924 23:06:31.771023 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.766607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56208 (local address 127.0.98.1:36095)
0924 23:06:31.766738 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.766743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.766914 (+ 171us) server_negotiation.cc:407] Connection header received
0924 23:06:31.766959 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.766961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.766975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.767045 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:31.767213 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.767909 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.769421 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.770138 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.770201 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.770252 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.770343 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.770387 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
I20260924 23:06:31.771368 4857 tablet_service.cc:1467] Processing CreateTablet for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b (DEFAULT_TABLE table=test-table [id=2afe65484dae4aa6a80da73388deb236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:31.771577 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.766336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36953 (local address 127.0.0.1:42564)
0924 23:06:31.766669 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.766677 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.766717 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.768353 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.768354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.768356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:31.768402 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.768403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.769592 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.769593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.770535 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.770537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.771380 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.771381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.771440 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.771444 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.771475 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.771510 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":74,"threads_started":1}
I20260924 23:06:31.771759 4857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:31.771685 4591 tablet_service.cc:1467] Processing CreateTablet for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b (DEFAULT_TABLE table=test-table [id=2afe65484dae4aa6a80da73388deb236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:31.771896 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.766495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.0.98.2:36953)
0924 23:06:31.768093 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.768098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.768106 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:31.768141 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.768143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.768158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.768344 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:31.768487 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.769499 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.770620 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.771277 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.771678 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.771739 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.771765 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.771810 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562,"thread_start_us":98,"threads_started":1}
I20260924 23:06:31.772017 4591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:31.774503 4937 tablet_bootstrap.cc:492] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Bootstrap starting.
I20260924 23:06:31.774591 4724 tablet_service.cc:1467] Processing CreateTablet for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b (DEFAULT_TABLE table=test-table [id=2afe65484dae4aa6a80da73388deb236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:31.774924 4724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:31.775171 4937 tablet_bootstrap.cc:654] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:31.775661 4937 log.cc:824] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:31.776028 4938 tablet_bootstrap.cc:492] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: Bootstrap starting.
I20260924 23:06:31.776492 4937 tablet_bootstrap.cc:492] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: No bootstrap required, opened a new log
I20260924 23:06:31.776571 4937 ts_tablet_manager.cc:1397] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:31.776568 4938 tablet_bootstrap.cc:654] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:31.777046 4938 log.cc:824] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:31.777846 4941 tablet_bootstrap.cc:492] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Bootstrap starting.
I20260924 23:06:31.778187 4938 tablet_bootstrap.cc:492] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: No bootstrap required, opened a new log
I20260924 23:06:31.778266 4938 ts_tablet_manager.cc:1397] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:31.778535 4941 tablet_bootstrap.cc:654] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:31.778523 4937 raft_consensus.cc:348] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.778718 4937 raft_consensus.cc:374] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:31.778761 4937 raft_consensus.cc:729] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cde94af7788482abb26297a2820a2ce, State: Initialized, Role: FOLLOWER
I20260924 23:06:31.778874 4937 consensus_queue.cc:260] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.779090 4941 log.cc:824] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:31.779181 4937 ts_tablet_manager.cc:1428] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:31.779304 4923 heartbeater.cc:503] Master 127.0.98.62:40457 was elected leader, sending a full tablet report...
I20260924 23:06:31.779800 4941 tablet_bootstrap.cc:492] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: No bootstrap required, opened a new log
I20260924 23:06:31.779874 4941 ts_tablet_manager.cc:1397] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:31.780506 4938 raft_consensus.cc:348] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.780687 4938 raft_consensus.cc:374] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:31.780731 4938 raft_consensus.cc:729] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709e13e1a7134235899dae64e69500e6, State: Initialized, Role: FOLLOWER
I20260924 23:06:31.780848 4938 consensus_queue.cc:260] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.781134 4938 ts_tablet_manager.cc:1428] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:31.781152 4657 heartbeater.cc:503] Master 127.0.98.62:40457 was elected leader, sending a full tablet report...
I20260924 23:06:31.781965 4941 raft_consensus.cc:348] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.782187 4941 raft_consensus.cc:374] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:31.782240 4941 raft_consensus.cc:729] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e02e7510a364797ba4a1ba33fb32fdd, State: Initialized, Role: FOLLOWER
I20260924 23:06:31.782346 4941 consensus_queue.cc:260] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.782586 4941 ts_tablet_manager.cc:1428] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:31.782617 4790 heartbeater.cc:503] Master 127.0.98.62:40457 was elected leader, sending a full tablet report...
W20260924 23:06:31.834901 4791 tablet.cc:2369] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:31.882058 4945 raft_consensus.cc:482] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:31.882176 4945 raft_consensus.cc:504] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.882508 4945 leader_election.cc:288] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cde94af7788482abb26297a2820a2ce (127.0.98.3:40595), 709e13e1a7134235899dae64e69500e6 (127.0.98.1:36095)
I20260924 23:06:31.886554 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.882725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37331 (local address 127.0.98.3:40595)
0924 23:06:31.882772 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.882775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.882785 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:31.882855 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.882855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.882869 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.882935 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:31.883119 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.883943 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.885574 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.886186 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.886263 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.886273 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.886440 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.886479 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:31.886554 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.882776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35631 (local address 127.0.98.1:36095)
0924 23:06:31.882827 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.882829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.883026 (+ 197us) server_negotiation.cc:407] Connection header received
0924 23:06:31.883055 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.883056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.883067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.883146 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:31.883281 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.883970 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.885575 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.886186 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.886263 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.886273 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.886440 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.886479 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:31.886569 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.882669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40595 (local address 127.0.98.2:37331)
0924 23:06:31.882726 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.882739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.882782 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.882956 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.882957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.882959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:31.883038 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.883040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.884053 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.884054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.885454 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.885456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.886274 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.886275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.886359 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.886362 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.886425 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.886469 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":112}
I20260924 23:06:31.886574 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.882700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36095 (local address 127.0.98.2:35631)
0924 23:06:31.882944 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.882954 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:31.883012 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:31.883163 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:31.883163 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:31.883164 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:31.883212 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.883213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.884053 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.884054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.885454 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:31.885456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.886274 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.886275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:31.886353 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.886358 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:31.886419 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:06:31.886469 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":63,"thread_start_us":126,"threads_started":1}
I20260924 23:06:31.886977 4877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60deb7e9249f4ab9a308b4a4f4a3f17b" candidate_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cde94af7788482abb26297a2820a2ce" is_pre_election: true
I20260924 23:06:31.887032 4611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60deb7e9249f4ab9a308b4a4f4a3f17b" candidate_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709e13e1a7134235899dae64e69500e6" is_pre_election: true
I20260924 23:06:31.887156 4877 raft_consensus.cc:2454] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e02e7510a364797ba4a1ba33fb32fdd in term 0.
I20260924 23:06:31.887200 4611 raft_consensus.cc:2454] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e02e7510a364797ba4a1ba33fb32fdd in term 0.
I20260924 23:06:31.887410 4679 leader_election.cc:302] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e02e7510a364797ba4a1ba33fb32fdd, 8cde94af7788482abb26297a2820a2ce; no voters:
I20260924 23:06:31.887562 4945 raft_consensus.cc:2781] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:31.887621 4945 raft_consensus.cc:482] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:31.887638 4945 raft_consensus.cc:3037] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:31.888356 4945 raft_consensus.cc:504] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.888507 4945 leader_election.cc:288] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [CANDIDATE]: Term 1 election: Requested vote from peers 8cde94af7788482abb26297a2820a2ce (127.0.98.3:40595), 709e13e1a7134235899dae64e69500e6 (127.0.98.1:36095)
I20260924 23:06:31.888670 4877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60deb7e9249f4ab9a308b4a4f4a3f17b" candidate_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cde94af7788482abb26297a2820a2ce"
I20260924 23:06:31.888731 4877 raft_consensus.cc:3037] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:31.889400 4877 raft_consensus.cc:2454] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e02e7510a364797ba4a1ba33fb32fdd in term 1.
I20260924 23:06:31.889521 4611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60deb7e9249f4ab9a308b4a4f4a3f17b" candidate_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709e13e1a7134235899dae64e69500e6"
I20260924 23:06:31.889583 4679 leader_election.cc:302] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e02e7510a364797ba4a1ba33fb32fdd, 8cde94af7788482abb26297a2820a2ce; no voters:
I20260924 23:06:31.889606 4611 raft_consensus.cc:3037] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:31.889683 4945 raft_consensus.cc:2781] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:31.889856 4945 raft_consensus.cc:686] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 1 LEADER]: Becoming Leader. State: Replica: 3e02e7510a364797ba4a1ba33fb32fdd, State: Running, Role: LEADER
I20260924 23:06:31.889977 4945 consensus_queue.cc:237] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } }
I20260924 23:06:31.890537 4611 raft_consensus.cc:2454] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e02e7510a364797ba4a1ba33fb32fdd in term 1.
I20260924 23:06:31.890803 4467 catalog_manager.cc:5495] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e02e7510a364797ba4a1ba33fb32fdd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e02e7510a364797ba4a1ba33fb32fdd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:31.915051 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.912522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.98.2:36953)
0924 23:06:31.912586 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:31.912588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:31.912597 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:31.912636 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:31.912637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:31.912665 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:31.912756 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:31.912935 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.913411 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.914240 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:31.914451 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:31.914525 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:31.914691 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:31.914803 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:31.914897 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:31.914919 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 23:06:31.914940 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:06:31.916790 4877 raft_consensus.cc:1261] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 1 FOLLOWER]: Refusing update from remote peer 3e02e7510a364797ba4a1ba33fb32fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:31.916844 4611 raft_consensus.cc:1261] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 1 FOLLOWER]: Refusing update from remote peer 3e02e7510a364797ba4a1ba33fb32fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:31.917161 4945 consensus_queue.cc:1035] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "709e13e1a7134235899dae64e69500e6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:31.917325 4947 consensus_queue.cc:1035] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cde94af7788482abb26297a2820a2ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:31.919505 4950 mvcc.cc:205] Tried to move back new op lower bound from 7333032722088505344 to 7333032721981825024. Current Snapshot: MvccSnapshot[applied={T|T < 7333032722088505344}]
I20260924 23:06:31.920644 4951 mvcc.cc:205] Tried to move back new op lower bound from 7333032722088505344 to 7333032721981825024. Current Snapshot: MvccSnapshot[applied={T|T < 7333032722088505344 or (T in {7333032722088505344})}]
I20260924 23:06:31.921829 4466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:31.921954 4466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:31.922688 4466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28c79734260f4be5a7e394fb44a4dd3d: Sending DeleteTablet for 3 replicas of tablet 60deb7e9249f4ab9a308b4a4f4a3f17b
I20260924 23:06:31.923026 4857 tablet_service.cc:1514] Processing DeleteTablet for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39358
I20260924 23:06:31.923025 4591 tablet_service.cc:1514] Processing DeleteTablet for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56208
I20260924 23:06:31.923251 4961 tablet_replica.cc:330] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6: stopping tablet replica
I20260924 23:06:31.923242 4724 tablet_service.cc:1514] Processing DeleteTablet for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42564
I20260924 23:06:31.923345 4961 raft_consensus.cc:2229] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:31.923445 4961 raft_consensus.cc:2258] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 709e13e1a7134235899dae64e69500e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:31.923473 4962 tablet_replica.cc:330] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: stopping tablet replica
I20260924 23:06:31.923703 4960 tablet_replica.cc:330] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: stopping tablet replica
I20260924 23:06:31.923779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4529
I20260924 23:06:31.923794 4960 raft_consensus.cc:2229] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:31.925045 4962 raft_consensus.cc:2229] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:31.925205 4962 raft_consensus.cc:2258] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:31.925668 4962 ts_tablet_manager.cc:1905] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:31.927779 4962 ts_tablet_manager.cc:1918] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:31.927906 4962 log.cc:1196] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/60deb7e9249f4ab9a308b4a4f4a3f17b
I20260924 23:06:31.928229 4962 ts_tablet_manager.cc:1939] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 3e02e7510a364797ba4a1ba33fb32fdd: Deleting consensus metadata
I20260924 23:06:31.928817 4452 catalog_manager.cc:4841] TS 3e02e7510a364797ba4a1ba33fb32fdd (127.0.98.2:36953): tablet 60deb7e9249f4ab9a308b4a4f4a3f17b (table test-table [id=2afe65484dae4aa6a80da73388deb236]) successfully deleted
W20260924 23:06:31.930114 4451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36095 (error 108)
I20260924 23:06:31.930294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4662
I20260924 23:06:31.931303 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:31.930845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36095 (local address 127.0.0.1:56214)
0924 23:06:31.931225 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:31.931263 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
W20260924 23:06:31.931468 4451 catalog_manager.cc:4584] TS 709e13e1a7134235899dae64e69500e6 (127.0.98.1:36095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60deb7e9249f4ab9a308b4a4f4a3f17b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36095: connect: Connection refused (error 111)
I20260924 23:06:31.932030 4960 raft_consensus.cc:2258] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:31.932987 4960 ts_tablet_manager.cc:1905] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:31.934926 4960 ts_tablet_manager.cc:1918] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:31.935000 4960 log.cc:1196] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/60deb7e9249f4ab9a308b4a4f4a3f17b
I20260924 23:06:31.935261 4960 ts_tablet_manager.cc:1939] T 60deb7e9249f4ab9a308b4a4f4a3f17b P 8cde94af7788482abb26297a2820a2ce: Deleting consensus metadata
I20260924 23:06:31.935915 4453 catalog_manager.cc:4841] TS 8cde94af7788482abb26297a2820a2ce (127.0.98.3:40595): tablet 60deb7e9249f4ab9a308b4a4f4a3f17b (table test-table [id=2afe65484dae4aa6a80da73388deb236]) successfully deleted
I20260924 23:06:31.936940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4795
I20260924 23:06:31.943450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4436
2026-09-24T23:06:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1975 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T23:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:31Z Disabled control of system clock
I20260924 23:06:31.969945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:06:32.049791 4967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:32.050060 4967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:32.050083 4967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:32.051211 4967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:32.051250 4967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:32.051263 4967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:32.051311 4967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:32.051328 4967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:32.052636 4967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:32.052901 4967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:32.053202 4967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:32.058027 4973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.058024 4972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.058260 4967 server_base.cc:1034] running on GCE node
W20260924 23:06:32.058018 4975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.058668 4967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:32.058960 4967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:32.060139 4967 hybrid_clock.cc:648] HybridClock initialized: now 1790291192060117 us; error 35 us; skew 500 ppm
I20260924 23:06:32.061352 4967 webserver.cc:460] Webserver started at http://127.0.98.62:42521/ using document root <none> and password file <none>
I20260924 23:06:32.061604 4967 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:32.061674 4967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:32.061827 4967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:32.062995 4967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "be616073262f4e6ab39813dab73853f5"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "d33c54ee3d16a2feb63f3212f62268d0"
server_key_iv: "467a1db539339c3bc0dd4aa7baf14970"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.063349 4967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "be616073262f4e6ab39813dab73853f5"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "d33c54ee3d16a2feb63f3212f62268d0"
server_key_iv: "467a1db539339c3bc0dd4aa7baf14970"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.064623 4967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:32.065271 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.065440 4967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:32.065544 4967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "be616073262f4e6ab39813dab73853f5"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "d33c54ee3d16a2feb63f3212f62268d0"
server_key_iv: "467a1db539339c3bc0dd4aa7baf14970"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.065616 4967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:32.068854 4967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:32.069088 4967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:32.069233 4967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:32.072774 4967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41347
I20260924 23:06:32.072824 5033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41347 every 8 connection(s)
I20260924 23:06:32.073156 4967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:32.073825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 4967
I20260924 23:06:32.073871 5034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:32.073943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:32.074136 392 external_mini_cluster.cc:1428] Setting key f9167ec4173c88d49c151838dc0842fa
I20260924 23:06:32.076745 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.074684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.0.98.62:41347)
0924 23:06:32.074890 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.074903 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.074924 (+ 21us) server_negotiation.cc:407] Connection header received
0924 23:06:32.075477 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.075483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.075522 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.075668 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.076327 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.076330 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.076330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.076334 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.076383 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.076393 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.076472 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.076539 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.076594 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260924 23:06:32.077512 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5: Bootstrap starting.
I20260924 23:06:32.078094 5034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:32.078312 5034 log.cc:824] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:32.078913 5034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5: No bootstrap required, opened a new log
I20260924 23:06:32.080135 5034 raft_consensus.cc:348] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be616073262f4e6ab39813dab73853f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260924 23:06:32.080259 5034 raft_consensus.cc:374] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:32.080281 5034 raft_consensus.cc:729] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be616073262f4e6ab39813dab73853f5, State: Initialized, Role: FOLLOWER
I20260924 23:06:32.080394 5034 consensus_queue.cc:260] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be616073262f4e6ab39813dab73853f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260924 23:06:32.080473 5034 raft_consensus.cc:388] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:32.080495 5034 raft_consensus.cc:482] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:32.080523 5034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:32.081012 5034 raft_consensus.cc:504] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be616073262f4e6ab39813dab73853f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260924 23:06:32.081089 5034 leader_election.cc:302] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be616073262f4e6ab39813dab73853f5; no voters:
I20260924 23:06:32.081204 5034 leader_election.cc:288] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:32.081434 5034 sys_catalog.cc:564] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:32.081544 5039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:32.081677 5039 raft_consensus.cc:686] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [term 1 LEADER]: Becoming Leader. State: Replica: be616073262f4e6ab39813dab73853f5, State: Running, Role: LEADER
I20260924 23:06:32.081787 5039 consensus_queue.cc:237] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be616073262f4e6ab39813dab73853f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } }
I20260924 23:06:32.082150 5039 sys_catalog.cc:455] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be616073262f4e6ab39813dab73853f5. Latest consensus state: current_term: 1 leader_uuid: "be616073262f4e6ab39813dab73853f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be616073262f4e6ab39813dab73853f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260924 23:06:32.082219 5039 sys_catalog.cc:458] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:32.082393 5048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:32.082872 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be616073262f4e6ab39813dab73853f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be616073262f4e6ab39813dab73853f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41347 } } }
I20260924 23:06:32.082958 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:32.082897 5048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:32.086927 5048 catalog_manager.cc:1347] Generated new cluster ID: a8d10292cca447acbc1e193630038814
I20260924 23:06:32.086992 5048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:32.113452 5048 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:32.113668 5048 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:32.121219 5048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5: Generated new TSK 0
I20260924 23:06:32.121467 5048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:06:32.125526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--builtin_ntp_servers=127.0.98.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:06:32.207759 5058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:32.207978 5058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:32.208001 5058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:32.209182 5058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:32.209220 5058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:32.209247 5058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:32.210738 5058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:32.211001 5058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:32.211298 5058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:32.216323 5064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.216358 5063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.216354 5066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.216502 5058 server_base.cc:1034] running on GCE node
I20260924 23:06:32.216725 5058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:32.216928 5058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:32.218062 5058 hybrid_clock.cc:648] HybridClock initialized: now 1790291192218050 us; error 28 us; skew 500 ppm
I20260924 23:06:32.219211 5058 webserver.cc:460] Webserver started at http://127.0.98.1:41989/ using document root <none> and password file <none>
I20260924 23:06:32.219446 5058 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:32.219491 5058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:32.219663 5058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:32.220757 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "d03553b0aee94b54896b4bb96a5ec5f9"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "83105043c5e02431bef4cb7ccabe690d"
server_key_iv: "f8e4532ff868c23209cc2260c2a0af4e"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.221122 5058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d03553b0aee94b54896b4bb96a5ec5f9"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "83105043c5e02431bef4cb7ccabe690d"
server_key_iv: "f8e4532ff868c23209cc2260c2a0af4e"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.222455 5058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:32.223150 5072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.223332 5058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:32.223419 5058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "d03553b0aee94b54896b4bb96a5ec5f9"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "83105043c5e02431bef4cb7ccabe690d"
server_key_iv: "f8e4532ff868c23209cc2260c2a0af4e"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.223492 5058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:32.226822 5058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:32.227092 5058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:32.227229 5058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:32.227437 5058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:32.227720 5058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:32.227751 5058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.227802 5058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:32.227829 5058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.233505 5058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41041
I20260924 23:06:32.233562 5185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41041 every 8 connection(s)
I20260924 23:06:32.233848 5058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:32.236717 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.234564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59185 (local address 127.0.98.62:41347)
0924 23:06:32.234649 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.234651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.234844 (+ 193us) server_negotiation.cc:407] Connection header received
0924 23:06:32.235326 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.235327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.235334 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.235391 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.236472 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.236474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.236474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.236479 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.236494 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.236498 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.236557 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.236634 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.236647 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:06:32.236899 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.234530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.1:59185)
0924 23:06:32.234719 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.234759 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.235220 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.235466 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.235470 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.235479 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.236327 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.236327 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.236353 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.236377 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.236557 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.236562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.236565 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.236642 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.236699 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260924 23:06:32.237347 5186 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260924 23:06:32.237438 5186 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:32.237540 5186 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260924 23:06:32.238076 4998 ts_manager.cc:194] Registered new tserver with Master: d03553b0aee94b54896b4bb96a5ec5f9 (127.0.98.1:41041)
I20260924 23:06:32.240329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5058
I20260924 23:06:32.240417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:32.240550 392 external_mini_cluster.cc:1428] Setting key a93a7a69efca0e1b94dee156e0944327
I20260924 23:06:32.241375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--builtin_ntp_servers=127.0.98.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:06:32.322626 5189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:32.322825 5189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:32.322847 5189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:32.323928 5189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:32.323961 5189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:32.323987 5189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:32.325285 5189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:32.325486 5189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:32.325788 5189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:32.330731 5195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.330725 5194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.330946 5189 server_base.cc:1034] running on GCE node
W20260924 23:06:32.331022 5197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.331243 5189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:32.331493 5189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:32.332633 5189 hybrid_clock.cc:648] HybridClock initialized: now 1790291192332616 us; error 33 us; skew 500 ppm
I20260924 23:06:32.333787 5189 webserver.cc:460] Webserver started at http://127.0.98.2:40095/ using document root <none> and password file <none>
I20260924 23:06:32.334080 5189 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:32.334152 5189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:32.334306 5189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:32.335345 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "aab22e4f7000404da0eeff1295248b05"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "9324aaff39ee1003fc391a1ee55e00e6"
server_key_iv: "759d4fae7b6701095923f67937f45625"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.335708 5189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aab22e4f7000404da0eeff1295248b05"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "9324aaff39ee1003fc391a1ee55e00e6"
server_key_iv: "759d4fae7b6701095923f67937f45625"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.337014 5189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:32.337738 5203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.337987 5189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:32.338083 5189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "aab22e4f7000404da0eeff1295248b05"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "9324aaff39ee1003fc391a1ee55e00e6"
server_key_iv: "759d4fae7b6701095923f67937f45625"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.338163 5189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:32.341424 5189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:32.341681 5189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:32.341823 5189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:32.342063 5189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:32.342357 5189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:32.342412 5189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.342453 5189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:32.342479 5189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.348007 5189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33845
I20260924 23:06:32.348042 5316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33845 every 8 connection(s)
I20260924 23:06:32.348349 5189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:32.351064 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.349038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42333 (local address 127.0.98.62:41347)
0924 23:06:32.349118 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.349121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.349296 (+ 175us) server_negotiation.cc:407] Connection header received
0924 23:06:32.349721 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.349722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.349727 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.349778 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.350862 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.350864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.350864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.350868 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.350880 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.350883 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.350951 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.351000 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.351012 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 23:06:32.351255 5319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.349005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.2:42333)
0924 23:06:32.349187 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.349217 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.349639 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.349827 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.349830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.349837 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.350733 (+ 896us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.350734 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.350760 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.350786 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.350952 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.350954 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.350956 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.351006 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.351051 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260924 23:06:32.351707 5317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260924 23:06:32.351791 5317 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:32.351904 5317 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260924 23:06:32.352221 4998 ts_manager.cc:194] Registered new tserver with Master: aab22e4f7000404da0eeff1295248b05 (127.0.98.2:33845)
I20260924 23:06:32.355803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5189
I20260924 23:06:32.355942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:32.356168 392 external_mini_cluster.cc:1428] Setting key b90e80d513c43a29d6133034cf742acc
I20260924 23:06:32.357053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--builtin_ntp_servers=127.0.98.20:40025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 23:06:32.436797 5320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:32.437000 5320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:32.437021 5320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:32.438117 5320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:32.438153 5320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:32.438179 5320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:32.439451 5320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:32.439635 5320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:32.439862 5320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:32.444845 5328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.444911 5326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.444852 5325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.445025 5320 server_base.cc:1034] running on GCE node
I20260924 23:06:32.445216 5320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:32.445410 5320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:32.446552 5320 hybrid_clock.cc:648] HybridClock initialized: now 1790291192446529 us; error 39 us; skew 500 ppm
I20260924 23:06:32.447921 5320 webserver.cc:460] Webserver started at http://127.0.98.3:38289/ using document root <none> and password file <none>
I20260924 23:06:32.448189 5320 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:32.448239 5320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:32.448415 5320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:32.449477 5320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "bf9a094cca74408db5ff387d17017dc7"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "14e4b2936f2da2b7b0201de73e800814"
server_key_iv: "054b20f2959fb5b06156e3c0ce03bd25"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.449852 5320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bf9a094cca74408db5ff387d17017dc7"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "14e4b2936f2da2b7b0201de73e800814"
server_key_iv: "054b20f2959fb5b06156e3c0ce03bd25"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.451212 5320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:32.452076 5334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.452260 5320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:32.452359 5320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "bf9a094cca74408db5ff387d17017dc7"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "14e4b2936f2da2b7b0201de73e800814"
server_key_iv: "054b20f2959fb5b06156e3c0ce03bd25"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.452445 5320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:32.455880 5320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:32.456164 5320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:32.456305 5320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:32.456517 5320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:32.456822 5320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:32.456880 5320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.456928 5320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:32.456956 5320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.462842 5320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39229
I20260924 23:06:32.462885 5447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39229 every 8 connection(s)
I20260924 23:06:32.463232 5320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:32.466444 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.463998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60681 (local address 127.0.98.62:41347)
0924 23:06:32.464086 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.464089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.464278 (+ 189us) server_negotiation.cc:407] Connection header received
0924 23:06:32.464795 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.464796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.464802 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.464861 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.466169 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.466170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.466171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.466176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.466192 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.466196 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.466249 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.466347 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.466371 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:06:32.466630 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.463955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41347 (local address 127.0.98.3:60681)
0924 23:06:32.464158 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.464191 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.464683 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.464948 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.464951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.464960 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.466013 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.466014 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.466043 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.466074 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.466246 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.466250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.466254 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.466344 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.466408 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260924 23:06:32.467187 5448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41347
I20260924 23:06:32.467267 5448 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:32.467394 5448 heartbeater.cc:511] Master 127.0.98.62:41347 requested a full tablet report, sending...
I20260924 23:06:32.467722 4998 ts_manager.cc:194] Registered new tserver with Master: bf9a094cca74408db5ff387d17017dc7 (127.0.98.3:39229)
I20260924 23:06:32.471580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5320
I20260924 23:06:32.471673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:32.471832 392 external_mini_cluster.cc:1428] Setting key 3ece98b94507889d9a0a37cd14aa223e
I20260924 23:06:32.472672 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:06:32.475068 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.474062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.98.62:41347)
0924 23:06:32.474123 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.474125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.474210 (+ 85us) server_negotiation.cc:407] Connection header received
0924 23:06:32.474251 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.474251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.474255 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.474303 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.474922 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.474922 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.474923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.474925 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.474935 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.474938 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.474978 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.475008 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.475018 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:32.476796 4998 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:32.478498 4998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:32.482715 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.480984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.0.98.2:33845)
0924 23:06:32.481662 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.481666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.481678 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:32.481704 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.481707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.481716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.482030 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.482407 (+ 377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.482412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.482412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.482416 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.482450 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.482468 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.482534 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.482620 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.482634 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":75,"threads_started":1}
I20260924 23:06:32.482705 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.480869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33845 (local address 127.0.0.1:52874)
0924 23:06:32.481120 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.481130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.481180 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.481817 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.481818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.481821 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.482282 (+ 461us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.482283 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.482291 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.482304 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.482529 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.482536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.482540 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.482610 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.482624 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260924 23:06:32.482800 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.480869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41041 (local address 127.0.0.1:49164)
0924 23:06:32.481075 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.481120 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.481180 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.481470 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.481477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.481484 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.482129 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.482129 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.482150 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.482176 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.482641 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.482641 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.482642 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.482673 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.482687 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
I20260924 23:06:32.483095 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.481172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.0.98.1:41041)
0924 23:06:32.481338 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.481342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.481351 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:32.481372 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.481373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.481381 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.481470 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.482501 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.482504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.482504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.482508 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.482539 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.482553 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.482972 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.483000 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.483016 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
I20260924 23:06:32.483340 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.480922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39229 (local address 127.0.0.1:53964)
0924 23:06:32.481174 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.481186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.481241 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.482019 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.482020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.482022 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.482421 (+ 399us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.482421 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.482426 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.482434 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.482641 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.482641 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.482642 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.483270 (+ 628us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.483284 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260924 23:06:32.483577 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.481382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.0.98.3:39229)
0924 23:06:32.481810 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.481815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.481825 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:32.481852 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.481853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.481862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.482018 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.482501 (+ 483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.482504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.482504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.482508 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.482539 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.482553 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.482616 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.482737 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.482752 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":119,"threads_started":1}
I20260924 23:06:32.483554 5251 tablet_service.cc:1467] Processing CreateTablet for tablet 1aca6b8a5dcd4b018ff6db4183168132 (DEFAULT_TABLE table=test-table [id=0cb1938c2e9a49059aa5cfb7be13b58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:32.483834 5120 tablet_service.cc:1467] Processing CreateTablet for tablet 1aca6b8a5dcd4b018ff6db4183168132 (DEFAULT_TABLE table=test-table [id=0cb1938c2e9a49059aa5cfb7be13b58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:32.483958 5251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aca6b8a5dcd4b018ff6db4183168132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:32.484206 5120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aca6b8a5dcd4b018ff6db4183168132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:32.484170 5382 tablet_service.cc:1467] Processing CreateTablet for tablet 1aca6b8a5dcd4b018ff6db4183168132 (DEFAULT_TABLE table=test-table [id=0cb1938c2e9a49059aa5cfb7be13b58e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:32.484496 5382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aca6b8a5dcd4b018ff6db4183168132. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:32.486857 5462 tablet_bootstrap.cc:492] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9: Bootstrap starting.
I20260924 23:06:32.487393 5463 tablet_bootstrap.cc:492] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Bootstrap starting.
I20260924 23:06:32.487843 5462 tablet_bootstrap.cc:654] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:32.487937 5464 tablet_bootstrap.cc:492] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Bootstrap starting.
I20260924 23:06:32.488058 5463 tablet_bootstrap.cc:654] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:32.488373 5463 log.cc:824] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:32.488373 5462 log.cc:824] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:32.488417 5464 tablet_bootstrap.cc:654] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:32.488691 5464 log.cc:824] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:32.489187 5462 tablet_bootstrap.cc:492] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9: No bootstrap required, opened a new log
I20260924 23:06:32.489187 5463 tablet_bootstrap.cc:492] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: No bootstrap required, opened a new log
I20260924 23:06:32.489251 5463 ts_tablet_manager.cc:1397] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:32.489251 5462 ts_tablet_manager.cc:1397] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:32.489372 5464 tablet_bootstrap.cc:492] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: No bootstrap required, opened a new log
I20260924 23:06:32.489415 5464 ts_tablet_manager.cc:1397] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:32.491607 5464 raft_consensus.cc:348] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.491786 5464 raft_consensus.cc:374] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:32.491830 5464 raft_consensus.cc:729] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9a094cca74408db5ff387d17017dc7, State: Initialized, Role: FOLLOWER
I20260924 23:06:32.491974 5464 consensus_queue.cc:260] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.491920 5462 raft_consensus.cc:348] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.491912 5463 raft_consensus.cc:348] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.492064 5462 raft_consensus.cc:374] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:32.492064 5463 raft_consensus.cc:374] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:32.492106 5462 raft_consensus.cc:729] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03553b0aee94b54896b4bb96a5ec5f9, State: Initialized, Role: FOLLOWER
I20260924 23:06:32.492107 5463 raft_consensus.cc:729] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab22e4f7000404da0eeff1295248b05, State: Initialized, Role: FOLLOWER
I20260924 23:06:32.492220 5463 consensus_queue.cc:260] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.492250 5464 ts_tablet_manager.cc:1428] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:32.492267 5462 consensus_queue.cc:260] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.492400 5448 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260924 23:06:32.492494 5462 ts_tablet_manager.cc:1428] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:32.492496 5463 ts_tablet_manager.cc:1428] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:06:32.492610 5186 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260924 23:06:32.492610 5317 heartbeater.cc:503] Master 127.0.98.62:41347 was elected leader, sending a full tablet report...
I20260924 23:06:32.564769 5470 raft_consensus.cc:482] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:32.564980 5470 raft_consensus.cc:504] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.565357 5470 leader_election.cc:288] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aab22e4f7000404da0eeff1295248b05 (127.0.98.2:33845), bf9a094cca74408db5ff387d17017dc7 (127.0.98.3:39229)
I20260924 23:06:32.566675 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.565597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45655 (local address 127.0.98.3:39229)
0924 23:06:32.565641 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.565643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.565653 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:32.565721 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.565722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.565727 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.565805 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.566411 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.566413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.566413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.566417 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.566435 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.566440 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.566523 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.566577 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.566593 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:32.566686 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.565546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39229 (local address 127.0.98.1:45655)
0924 23:06:32.565592 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.565605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.565649 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.565809 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.565809 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.565812 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.566287 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.566288 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.566294 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.566304 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.566527 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.566527 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.566528 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.566593 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.566608 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 23:06:32.566740 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.565596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52781 (local address 127.0.98.2:33845)
0924 23:06:32.565646 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.565649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.565868 (+ 219us) server_negotiation.cc:407] Connection header received
0924 23:06:32.565974 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.565975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.565980 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.566033 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.566513 (+ 480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.566515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.566515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.566519 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.566537 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.566542 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.566598 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.566652 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.566664 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:32.567092 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.565550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33845 (local address 127.0.98.1:52781)
0924 23:06:32.565788 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.565803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.565877 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.566048 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.566049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.566052 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:32.566418 (+ 366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:06:32.566418 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:32.566423 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:06:32.566432 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:32.566604 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:32.566605 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:32.566605 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 23:06:32.566652 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:06:32.566662 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":71,"thread_start_us":113,"threads_started":1}
I20260924 23:06:32.567067 5402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca6b8a5dcd4b018ff6db4183168132" candidate_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9a094cca74408db5ff387d17017dc7" is_pre_election: true
I20260924 23:06:32.567210 5402 raft_consensus.cc:2454] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03553b0aee94b54896b4bb96a5ec5f9 in term 0.
I20260924 23:06:32.567447 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca6b8a5dcd4b018ff6db4183168132" candidate_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab22e4f7000404da0eeff1295248b05" is_pre_election: true
I20260924 23:06:32.567487 5073 leader_election.cc:302] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf9a094cca74408db5ff387d17017dc7, d03553b0aee94b54896b4bb96a5ec5f9; no voters:
I20260924 23:06:32.567605 5271 raft_consensus.cc:2454] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03553b0aee94b54896b4bb96a5ec5f9 in term 0.
I20260924 23:06:32.567626 5470 raft_consensus.cc:2781] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:32.567668 5470 raft_consensus.cc:482] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:32.567700 5470 raft_consensus.cc:3037] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:32.568471 5470 raft_consensus.cc:504] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.568608 5470 leader_election.cc:288] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [CANDIDATE]: Term 1 election: Requested vote from peers aab22e4f7000404da0eeff1295248b05 (127.0.98.2:33845), bf9a094cca74408db5ff387d17017dc7 (127.0.98.3:39229)
I20260924 23:06:32.568748 5402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca6b8a5dcd4b018ff6db4183168132" candidate_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9a094cca74408db5ff387d17017dc7"
I20260924 23:06:32.568748 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aca6b8a5dcd4b018ff6db4183168132" candidate_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aab22e4f7000404da0eeff1295248b05"
I20260924 23:06:32.568827 5271 raft_consensus.cc:3037] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:32.568825 5402 raft_consensus.cc:3037] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:32.569693 5402 raft_consensus.cc:2454] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03553b0aee94b54896b4bb96a5ec5f9 in term 1.
I20260924 23:06:32.569693 5271 raft_consensus.cc:2454] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03553b0aee94b54896b4bb96a5ec5f9 in term 1.
I20260924 23:06:32.569866 5076 leader_election.cc:302] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aab22e4f7000404da0eeff1295248b05, d03553b0aee94b54896b4bb96a5ec5f9; no voters:
I20260924 23:06:32.570003 5470 raft_consensus.cc:2781] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:32.570148 5470 raft_consensus.cc:686] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [term 1 LEADER]: Becoming Leader. State: Replica: d03553b0aee94b54896b4bb96a5ec5f9, State: Running, Role: LEADER
I20260924 23:06:32.570253 5470 consensus_queue.cc:237] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } }
I20260924 23:06:32.571097 4997 catalog_manager.cc:5495] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d03553b0aee94b54896b4bb96a5ec5f9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d03553b0aee94b54896b4bb96a5ec5f9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } health_report { overall_health: HEALTHY } } }
I20260924 23:06:32.579502 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.578436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.0.98.1:41041)
0924 23:06:32.578492 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.578494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.578502 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:32.578559 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.578560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.578575 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.578633 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.579264 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.579266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.579266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.579270 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.579286 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.579291 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.579344 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.579411 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.579426 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:06:32.581364 5271 raft_consensus.cc:1261] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 1 FOLLOWER]: Refusing update from remote peer d03553b0aee94b54896b4bb96a5ec5f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:32.581372 5402 raft_consensus.cc:1261] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 1 FOLLOWER]: Refusing update from remote peer d03553b0aee94b54896b4bb96a5ec5f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:32.581738 5470 consensus_queue.cc:1035] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab22e4f7000404da0eeff1295248b05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:32.581933 5472 consensus_queue.cc:1035] T 1aca6b8a5dcd4b018ff6db4183168132 P d03553b0aee94b54896b4bb96a5ec5f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9a094cca74408db5ff387d17017dc7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:32.584955 5477 mvcc.cc:205] Tried to move back new op lower bound from 7333032724810145792 to 7333032724768440320. Current Snapshot: MvccSnapshot[applied={T|T < 7333032724810145792}]
I20260924 23:06:32.586233 4997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:32.586323 4997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:32.587036 4997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be616073262f4e6ab39813dab73853f5: Sending DeleteTablet for 3 replicas of tablet 1aca6b8a5dcd4b018ff6db4183168132
I20260924 23:06:32.587409 5382 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aca6b8a5dcd4b018ff6db4183168132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:32 UTC) from {username='slave'} at 127.0.0.1:53964
I20260924 23:06:32.587389 5251 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aca6b8a5dcd4b018ff6db4183168132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:32 UTC) from {username='slave'} at 127.0.0.1:52874
I20260924 23:06:32.587625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5058
I20260924 23:06:32.587713 5120 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aca6b8a5dcd4b018ff6db4183168132 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:32 UTC) from {username='slave'} at 127.0.0.1:49164
I20260924 23:06:32.588977 5485 tablet_replica.cc:330] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: stopping tablet replica
I20260924 23:06:32.589090 5485 raft_consensus.cc:2229] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:32.589198 5485 raft_consensus.cc:2258] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:32.589347 5486 tablet_replica.cc:330] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: stopping tablet replica
I20260924 23:06:32.589445 5486 raft_consensus.cc:2229] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:32.589545 5486 raft_consensus.cc:2258] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:32.589998 5485 ts_tablet_manager.cc:1905] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:32.592032 5485 ts_tablet_manager.cc:1918] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:32.592108 5485 log.cc:1196] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/1aca6b8a5dcd4b018ff6db4183168132
I20260924 23:06:32.592276 5486 ts_tablet_manager.cc:1905] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:32.592391 5485 ts_tablet_manager.cc:1939] T 1aca6b8a5dcd4b018ff6db4183168132 P bf9a094cca74408db5ff387d17017dc7: Deleting consensus metadata
I20260924 23:06:32.592844 4982 catalog_manager.cc:4841] TS bf9a094cca74408db5ff387d17017dc7 (127.0.98.3:39229): tablet 1aca6b8a5dcd4b018ff6db4183168132 (table test-table [id=0cb1938c2e9a49059aa5cfb7be13b58e]) successfully deleted
W20260924 23:06:32.593920 4983 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41041 (error 108)
I20260924 23:06:32.593963 5486 ts_tablet_manager.cc:1918] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:32.594027 5486 log.cc:1196] T 1aca6b8a5dcd4b018ff6db4183168132 P aab22e4f7000404da0eeff1295248b05: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/1aca6b8a5dcd4b018ff6db4183168132
I20260924 23:06:32.594132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5189
I20260924 23:06:32.594702 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.594286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41041 (local address 127.0.0.1:49190)
0924 23:06:32.594619 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.594662 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
W20260924 23:06:32.595698 4983 catalog_manager.cc:4584] TS d03553b0aee94b54896b4bb96a5ec5f9 (127.0.98.1:41041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1aca6b8a5dcd4b018ff6db4183168132: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41041: connect: Connection refused (error 111)
I20260924 23:06:32.599079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5320
I20260924 23:06:32.599102 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.598974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33845 (local address 127.0.0.1:52890)
0924 23:06:32.599033 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.599065 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:06:32.604012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 4967
2026-09-24T23:06:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (661 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 23:06:32.625026 392 mini_oidc.cc:152] Starting JWKS server
I20260924 23:06:32.626307 392 webserver.cc:460] Webserver started at https://127.0.0.1:40617/ using document root <none> and password file <none>
I20260924 23:06:32.626437 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 23:06:32.626627 392 webserver.cc:460] Webserver started at http://127.0.0.1:40295/ using document root <none> and password file <none>
2026-09-24T23:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:32Z Disabled control of system clock
I20260924 23:06:32.635361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45557
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40617/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/testchainca.cert with env {}
W20260924 23:06:32.715530 5493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:32.715762 5493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:32.715785 5493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:32.716969 5493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:32.717007 5493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:32.717020 5493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:32.717072 5493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:32.717096 5493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:06:32.717120 5493 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:06:32.717141 5493 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40617/jwks.json
I20260924 23:06:32.718724 5493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40617/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:32.718974 5493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:32.719192 5493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:32.724002 5501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.724086 5493 server_base.cc:1034] running on GCE node
W20260924 23:06:32.723968 5498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:32.723973 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:32.724416 5493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:32.724691 5493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:32.725831 5493 hybrid_clock.cc:648] HybridClock initialized: now 1790291192725805 us; error 40 us; skew 500 ppm
I20260924 23:06:32.727139 5493 webserver.cc:460] Webserver started at http://127.0.98.62:36677/ using document root <none> and password file <none>
I20260924 23:06:32.727382 5493 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:32.727432 5493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:32.727519 5493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:32.728483 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "67da445ad30343fe952d90a08f329d84"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "f6ee1dee7299033862e526b60b190905"
server_key_iv: "de47cbd16999a18a1930b9894e811e80"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.728777 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "67da445ad30343fe952d90a08f329d84"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "f6ee1dee7299033862e526b60b190905"
server_key_iv: "de47cbd16999a18a1930b9894e811e80"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.730022 5493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:32.730674 5507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:32.730898 5493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:32.730966 5493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "67da445ad30343fe952d90a08f329d84"
format_stamp: "Formatted at 2026-09-24 23:06:32 on dist-test-slave-ph9j"
server_key: "f6ee1dee7299033862e526b60b190905"
server_key_iv: "de47cbd16999a18a1930b9894e811e80"
server_key_version: "encryptionkey@0"
I20260924 23:06:32.731015 5493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:32.748034 5493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:32.748308 5493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:32.748410 5493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:32.752619 5493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45557
I20260924 23:06:32.752656 5559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45557 every 8 connection(s)
I20260924 23:06:32.753036 5493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:32.754101 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:32.756631 5560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84: Bootstrap starting.
I20260924 23:06:32.757222 5560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:32.757512 5560 log.cc:824] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:32.758317 5560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84: No bootstrap required, opened a new log
I20260924 23:06:32.759671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5493
I20260924 23:06:32.759760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:32.759948 392 external_mini_cluster.cc:1428] Setting key dcc437c458b3291248cf0c9c2133232f
I20260924 23:06:32.760118 5560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da445ad30343fe952d90a08f329d84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45557 } }
I20260924 23:06:32.760284 5560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:32.760329 5560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67da445ad30343fe952d90a08f329d84, State: Initialized, Role: FOLLOWER
I20260924 23:06:32.760486 5560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da445ad30343fe952d90a08f329d84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45557 } }
I20260924 23:06:32.760586 5560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:32.760643 5560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:32.760694 5560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:32.765188 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.760436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.0.98.62:45557)
0924 23:06:32.760658 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.760668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.760704 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:06:32.760821 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.760827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.760878 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.761031 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:32.761241 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:32.762509 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:32.763321 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:32.763722 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:32.763854 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:32.764533 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:32.764543 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:32.764544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:32.764705 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:32.764739 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:32.764749 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:32.764830 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:32.764936 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 23:06:32.765022 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260924 23:06:32.766286 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:06:32.826754 5560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da445ad30343fe952d90a08f329d84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45557 } }
I20260924 23:06:32.826980 5560 leader_election.cc:302] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67da445ad30343fe952d90a08f329d84; no voters:
I20260924 23:06:32.827212 5560 leader_election.cc:288] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:32.827306 5565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:32.827489 5565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [term 1 LEADER]: Becoming Leader. State: Replica: 67da445ad30343fe952d90a08f329d84, State: Running, Role: LEADER
I20260924 23:06:32.827553 5560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:32.827673 5565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da445ad30343fe952d90a08f329d84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45557 } }
I20260924 23:06:32.828145 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67da445ad30343fe952d90a08f329d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da445ad30343fe952d90a08f329d84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45557 } } }
I20260924 23:06:32.828238 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:32.828373 5567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67da445ad30343fe952d90a08f329d84. Latest consensus state: current_term: 1 leader_uuid: "67da445ad30343fe952d90a08f329d84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67da445ad30343fe952d90a08f329d84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45557 } } }
I20260924 23:06:32.828454 5567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:32.828585 5574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:32.829275 5574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:32.832779 5574 catalog_manager.cc:1347] Generated new cluster ID: e522ea6582f944baa70f1740f90b2a53
I20260924 23:06:32.832846 5574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:32.843917 5574 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:32.844646 5574 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:32.863055 5574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67da445ad30343fe952d90a08f329d84: Generated new TSK 0
I20260924 23:06:32.863297 5574 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 23:06:32.898940 5590 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 23:06:32.899055 5590 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 23:06:32.899374 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:32.883415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53062 (local address 127.0.98.62:45557)
0924 23:06:32.883511 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:32.883513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:32.883731 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:06:32.883822 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:32.883823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:32.883836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:32.883921 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:06:32.884106 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:32.884878 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:32.885716 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:32.886046 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:32.886118 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:32.886301 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:06:32.899095 (+ 12794us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 23:06:32.899292 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 23:06:32.899434 5564 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53062: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 23:06:32.899703 5589 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:32.883344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45557 (local address 127.0.0.1:53062)
0924 23:06:32.883642 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:32.883653 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:32.883728 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:32.883963 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:32.883964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:32.883966 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:06:32.884032 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:32.884034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:32.884983 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:32.884984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:32.885633 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:32.885636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:32.886117 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:32.886121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:32.886168 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:32.886186 (+ 18us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:06:32.899250 (+ 13064us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 23:06:32.899356 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":176,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40617/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 23:06:32.900377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5493
2026-09-24T23:06:32Z chronyd exiting
I20260924 23:06:32.936488 392 test_util.cc:175] -----------------------------------------------
I20260924 23:06:32.936564 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790291169445806-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 23:06:32.937294 392 mini_oidc.cc:152] Starting JWKS server
I20260924 23:06:32.938045 392 webserver.cc:460] Webserver started at https://127.0.0.1:38141/ using document root <none> and password file <none>
I20260924 23:06:32.938087 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 23:06:32.938268 392 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
2026-09-24T23:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:32Z Disabled control of system clock
I20260924 23:06:32.945551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46489
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/testchainca.cert with env {}
W20260924 23:06:33.024837 5596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:33.025053 5596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:33.025075 5596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:33.026271 5596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:33.026315 5596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:33.026328 5596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:33.026341 5596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:33.026353 5596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:06:33.026366 5596 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:06:33.026377 5596 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38141/jwks.json
I20260924 23:06:33.027840 5596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:33.028049 5596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:33.028218 5596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:33.033105 5602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:33.033161 5596 server_base.cc:1034] running on GCE node
W20260924 23:06:33.033216 5604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:33.033124 5601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:33.033605 5596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:33.033859 5596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:33.035048 5596 hybrid_clock.cc:648] HybridClock initialized: now 1790291193035027 us; error 39 us; skew 500 ppm
I20260924 23:06:33.036214 5596 webserver.cc:460] Webserver started at http://127.0.98.62:34367/ using document root <none> and password file <none>
I20260924 23:06:33.036458 5596 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:33.036501 5596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:33.036749 5596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:33.037869 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "5a4040c7789142f58899b56634fc42e4"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "6eeb06cfedf837668b96540c2c30f383"
server_key_iv: "b9c98bcbd4b41ab097c866773ff53067"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.038338 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a4040c7789142f58899b56634fc42e4"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "6eeb06cfedf837668b96540c2c30f383"
server_key_iv: "b9c98bcbd4b41ab097c866773ff53067"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.039664 5596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:33.040341 5610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:33.040517 5596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:33.040630 5596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "5a4040c7789142f58899b56634fc42e4"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "6eeb06cfedf837668b96540c2c30f383"
server_key_iv: "b9c98bcbd4b41ab097c866773ff53067"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.040702 5596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:33.054929 5596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:33.055251 5596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:33.055405 5596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:33.059231 5596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46489
I20260924 23:06:33.059279 5662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46489 every 8 connection(s)
I20260924 23:06:33.059629 5596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:33.060500 5663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:33.062780 5663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4: Bootstrap starting.
I20260924 23:06:33.063331 5663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:33.063620 5663 log.cc:824] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:33.064327 5663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4: No bootstrap required, opened a new log
I20260924 23:06:33.065857 5663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4040c7789142f58899b56634fc42e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46489 } }
I20260924 23:06:33.066084 5663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:33.066134 5663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4040c7789142f58899b56634fc42e4, State: Initialized, Role: FOLLOWER
I20260924 23:06:33.066309 5663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4040c7789142f58899b56634fc42e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46489 } }
I20260924 23:06:33.066422 5663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:33.066466 5663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:33.066519 5663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:33.067014 5663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4040c7789142f58899b56634fc42e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46489 } }
I20260924 23:06:33.067129 5663 leader_election.cc:302] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a4040c7789142f58899b56634fc42e4; no voters:
I20260924 23:06:33.067296 5663 leader_election.cc:288] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:33.067338 5666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:33.067516 5666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4040c7789142f58899b56634fc42e4, State: Running, Role: LEADER
I20260924 23:06:33.067559 5663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:33.067668 5666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4040c7789142f58899b56634fc42e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46489 } }
I20260924 23:06:33.068059 5667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a4040c7789142f58899b56634fc42e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4040c7789142f58899b56634fc42e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46489 } } }
I20260924 23:06:33.068149 5667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:33.068171 5668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a4040c7789142f58899b56634fc42e4. Latest consensus state: current_term: 1 leader_uuid: "5a4040c7789142f58899b56634fc42e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a4040c7789142f58899b56634fc42e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46489 } } }
I20260924 23:06:33.068266 5668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:33.068436 5675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:33.068914 5675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:33.069638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5596
I20260924 23:06:33.069756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:33.069980 392 external_mini_cluster.cc:1428] Setting key 44c12ce5c7d21d4ca1bc7e26061ad9a9
I20260924 23:06:33.070765 5675 catalog_manager.cc:1347] Generated new cluster ID: 944f039a0aee4c38b3c5e3c06190bb91
I20260924 23:06:33.070822 5675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:33.075980 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:33.070411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.98.62:46489)
0924 23:06:33.070754 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:33.070762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:33.070792 (+ 30us) server_negotiation.cc:407] Connection header received
0924 23:06:33.070919 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:33.070926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:33.070997 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:33.071123 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:33.071363 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.072667 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.073800 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.074435 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.074542 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.075286 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:33.075290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:33.075290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:33.075489 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:33.075528 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:33.075541 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:33.075602 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:33.075700 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 23:06:33.075772 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":63,"threads_started":1}
I20260924 23:06:33.077172 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:06:33.084342 5675 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:33.084861 5675 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:33.093797 5675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a4040c7789142f58899b56634fc42e4: Generated new TSK 0
I20260924 23:06:33.094054 5675 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 23:06:33.114980 5693 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 23:06:33.115077 5693 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 23:06:33.115398 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:33.100994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.98.62:46489)
0924 23:06:33.101037 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:33.101039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:33.101111 (+ 72us) server_negotiation.cc:407] Connection header received
0924 23:06:33.101191 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:33.101191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:33.101201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:33.101260 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:06:33.101380 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.101937 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.103069 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.103267 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.103317 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.103502 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:06:33.115215 (+ 11713us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 23:06:33.115341 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 23:06:33.115474 5686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36588: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 23:06:33.115563 5692 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:33.100887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46489 (local address 127.0.0.1:36588)
0924 23:06:33.101041 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:33.101053 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:33.101122 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:33.101261 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:33.101262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:33.101264 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:06:33.101322 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:33.101324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.102017 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.102018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:33.102983 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:33.102986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.103336 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.103339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:33.103406 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.103412 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:06:33.115337 (+ 11925us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 23:06:33.115470 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46489: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38141/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260924 23:06:33.115999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5596
2026-09-24T23:06:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (216 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 23:06:33.153632 392 mini_oidc.cc:152] Starting JWKS server
I20260924 23:06:33.154399 392 webserver.cc:460] Webserver started at https://127.0.0.1:35929/ using document root <none> and password file <none>
I20260924 23:06:33.154470 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 23:06:33.154631 392 webserver.cc:460] Webserver started at http://127.0.0.1:44161/ using document root <none> and password file <none>
2026-09-24T23:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:33Z Disabled control of system clock
I20260924 23:06:33.162062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35439
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35439
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35929/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 23:06:33.239701 5699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:33.239907 5699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:33.239928 5699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:33.240983 5699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:33.241016 5699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:33.241029 5699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:33.241042 5699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:33.241055 5699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:06:33.241066 5699 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:06:33.241078 5699 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35929/jwks.json
I20260924 23:06:33.242354 5699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35929/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:33.242530 5699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:33.242744 5699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:33.247419 5705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:33.247481 5707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:33.247432 5704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:33.247649 5699 server_base.cc:1034] running on GCE node
I20260924 23:06:33.247882 5699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:33.248131 5699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:33.249275 5699 hybrid_clock.cc:648] HybridClock initialized: now 1790291193249262 us; error 29 us; skew 500 ppm
I20260924 23:06:33.259430 5699 webserver.cc:460] Webserver started at http://127.0.98.62:33713/ using document root <none> and password file <none>
I20260924 23:06:33.259685 5699 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:33.259734 5699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:33.259897 5699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:33.260983 5699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "dc6763de2369447286fe80f154194fd2"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "d46f4df7585dc7443483d98bcf95cf1b"
server_key_iv: "f24a56c83afc089f14f5bd6aea103b92"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.261354 5699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "dc6763de2369447286fe80f154194fd2"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "d46f4df7585dc7443483d98bcf95cf1b"
server_key_iv: "f24a56c83afc089f14f5bd6aea103b92"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.262737 5699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:33.263494 5713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:33.263667 5699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:33.263751 5699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "dc6763de2369447286fe80f154194fd2"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "d46f4df7585dc7443483d98bcf95cf1b"
server_key_iv: "f24a56c83afc089f14f5bd6aea103b92"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.263803 5699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:33.275440 5699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:33.275704 5699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:33.275840 5699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:33.279569 5699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35439
I20260924 23:06:33.279620 5765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35439 every 8 connection(s)
I20260924 23:06:33.279946 5699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:33.280627 5766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:33.283190 5766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2: Bootstrap starting.
I20260924 23:06:33.283715 5766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:33.283952 5766 log.cc:824] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:33.284569 5766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2: No bootstrap required, opened a new log
I20260924 23:06:33.286100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5699
I20260924 23:06:33.286211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:33.286209 5766 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6763de2369447286fe80f154194fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35439 } }
I20260924 23:06:33.286383 5766 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:33.286409 392 external_mini_cluster.cc:1428] Setting key fe4567dd7277ed6e1ea9f3a1e5bfe531
I20260924 23:06:33.286419 5766 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc6763de2369447286fe80f154194fd2, State: Initialized, Role: FOLLOWER
I20260924 23:06:33.286577 5766 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6763de2369447286fe80f154194fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35439 } }
I20260924 23:06:33.286644 5766 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:33.286698 5766 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:33.287050 5766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:33.287710 5766 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6763de2369447286fe80f154194fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35439 } }
I20260924 23:06:33.287796 5766 leader_election.cc:302] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc6763de2369447286fe80f154194fd2; no voters:
I20260924 23:06:33.287971 5766 leader_election.cc:288] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:33.288004 5771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:33.288106 5771 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [term 1 LEADER]: Becoming Leader. State: Replica: dc6763de2369447286fe80f154194fd2, State: Running, Role: LEADER
I20260924 23:06:33.288221 5766 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:33.288235 5771 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6763de2369447286fe80f154194fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35439 } }
I20260924 23:06:33.288645 5771 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc6763de2369447286fe80f154194fd2. Latest consensus state: current_term: 1 leader_uuid: "dc6763de2369447286fe80f154194fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6763de2369447286fe80f154194fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35439 } } }
I20260924 23:06:33.288736 5771 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:33.288807 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc6763de2369447286fe80f154194fd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc6763de2369447286fe80f154194fd2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35439 } } }
I20260924 23:06:33.288965 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:33.290077 5785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:33.290592 5785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:33.291990 5785 catalog_manager.cc:1347] Generated new cluster ID: d1aef81dfa7049249076d4928cbaf313
I20260924 23:06:33.292042 5785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:33.292570 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:33.286852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.0.98.62:35439)
0924 23:06:33.287087 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:33.287094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:33.287145 (+ 51us) server_negotiation.cc:407] Connection header received
0924 23:06:33.287249 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:33.287255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:33.287305 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:33.287422 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:33.287647 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.289400 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.290412 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.290988 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.291109 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.291869 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:33.291873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:33.291873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:33.292079 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:33.292119 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:33.292132 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:33.292197 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:33.292291 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:06:33.292366 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260924 23:06:33.293478 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:06:33.297240 5785 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:33.297719 5785 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:33.301501 5785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc6763de2369447286fe80f154194fd2: Generated new TSK 0
I20260924 23:06:33.301666 5785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:06:33.311163 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:33.307173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.0.98.62:35439)
0924 23:06:33.307210 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:33.307212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:33.307344 (+ 132us) server_negotiation.cc:407] Connection header received
0924 23:06:33.307419 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:33.307420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:33.307430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:33.307488 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:06:33.307620 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.308079 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.309209 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.309382 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.309432 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.309603 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:06:33.310968 (+ 1365us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 23:06:33.311092 (+ 124us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":14}
W20260924 23:06:33.311223 5770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 23:06:33.311544 5795 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:33.307075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35439 (local address 127.0.0.1:55194)
0924 23:06:33.307268 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:33.307279 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:33.307346 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:33.307504 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:33.307505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:33.307507 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:06:33.307561 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:33.307563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.308167 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.308168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:33.309126 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:33.309129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.309451 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.309455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:33.309514 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.309519 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:06:33.311072 (+ 1553us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 23:06:33.311178 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35439: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35929/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260924 23:06:33.311758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5699
E20260924 23:06:33.323751 5796 webserver.cc:533] Webserver: sslize failed:
E20260924 23:06:33.323844 5796 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T23:06:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (196 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T23:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:33Z Disabled control of system clock
I20260924 23:06:33.356971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45071
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45071
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/testchainca.cert with env {}
W20260924 23:06:33.442270 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:33.442483 5800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:33.442505 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:33.443624 5800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:33.443660 5800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:33.443672 5800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:33.443686 5800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:33.443697 5800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 23:06:33.443709 5800 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 23:06:33.443720 5800 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 23:06:33.445171 5800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:33.445369 5800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:33.445530 5800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:33.450407 5806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:33.450407 5808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:33.450531 5805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:33.450716 5800 server_base.cc:1034] running on GCE node
I20260924 23:06:33.450997 5800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:33.451241 5800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:33.452387 5800 hybrid_clock.cc:648] HybridClock initialized: now 1790291193452372 us; error 28 us; skew 500 ppm
I20260924 23:06:33.453727 5800 webserver.cc:460] Webserver started at http://127.0.98.62:32875/ using document root <none> and password file <none>
I20260924 23:06:33.454032 5800 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:33.454083 5800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:33.454229 5800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:33.455329 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "1cf3c29338194299ad403b3dbd2686f4"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "c01488dc6581ab26125616291be3318d"
server_key_iv: "a9a3c39624fa2bc22ca2b844f37f2cda"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.455687 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cf3c29338194299ad403b3dbd2686f4"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "c01488dc6581ab26125616291be3318d"
server_key_iv: "a9a3c39624fa2bc22ca2b844f37f2cda"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.457019 5800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:33.457844 5814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:33.458164 5800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:33.458271 5800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "1cf3c29338194299ad403b3dbd2686f4"
format_stamp: "Formatted at 2026-09-24 23:06:33 on dist-test-slave-ph9j"
server_key: "c01488dc6581ab26125616291be3318d"
server_key_iv: "a9a3c39624fa2bc22ca2b844f37f2cda"
server_key_version: "encryptionkey@0"
I20260924 23:06:33.458359 5800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:33.481541 5800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:33.481941 5800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:33.482124 5800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:33.486449 5800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45071
I20260924 23:06:33.486497 5866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45071 every 8 connection(s)
I20260924 23:06:33.486920 5800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:33.487885 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:33.490247 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4: Bootstrap starting.
I20260924 23:06:33.490831 5867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:33.491124 5867 log.cc:824] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:33.491633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5800
I20260924 23:06:33.491736 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:33.491940 392 external_mini_cluster.cc:1428] Setting key ea3ea2f64fab810c387c3c0331c91ba7
I20260924 23:06:33.491935 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4: No bootstrap required, opened a new log
I20260924 23:06:33.494562 5867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf3c29338194299ad403b3dbd2686f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260924 23:06:33.494746 5867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:33.494786 5867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf3c29338194299ad403b3dbd2686f4, State: Initialized, Role: FOLLOWER
I20260924 23:06:33.494925 5867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf3c29338194299ad403b3dbd2686f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260924 23:06:33.495004 5867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:33.495035 5867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:33.495080 5867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:33.495769 5867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf3c29338194299ad403b3dbd2686f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260924 23:06:33.495854 5867 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf3c29338194299ad403b3dbd2686f4; no voters:
I20260924 23:06:33.496080 5867 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:33.496099 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:33.496245 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf3c29338194299ad403b3dbd2686f4, State: Running, Role: LEADER
I20260924 23:06:33.496345 5867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:33.496327 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf3c29338194299ad403b3dbd2686f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } }
I20260924 23:06:33.496805 5872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf3c29338194299ad403b3dbd2686f4. Latest consensus state: current_term: 1 leader_uuid: "1cf3c29338194299ad403b3dbd2686f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf3c29338194299ad403b3dbd2686f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } }
I20260924 23:06:33.496886 5872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:33.497041 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf3c29338194299ad403b3dbd2686f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf3c29338194299ad403b3dbd2686f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45071 } } }
I20260924 23:06:33.497169 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:33.497334 5880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:33.497860 5880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:33.498163 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:33.492470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.0.98.62:45071)
0924 23:06:33.492670 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:33.492677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:33.492701 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:06:33.493252 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:33.493259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:33.493316 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:33.493435 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:33.493658 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.494576 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.495646 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.496238 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.496673 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.497111 (+ 438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:33.497115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:33.497115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:06:33.497322 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:33.497365 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:06:33.497380 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:33.497677 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:33.497709 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:06:33.497782 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260924 23:06:33.499130 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:06:33.499852 5880 catalog_manager.cc:1347] Generated new cluster ID: e911bb586a024b68bae516fc78a77693
I20260924 23:06:33.499902 5880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:33.518719 5880 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:33.519428 5880 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:33.542317 5880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf3c29338194299ad403b3dbd2686f4: Generated new TSK 0
I20260924 23:06:33.542548 5880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:06:33.639339 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:33.551423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.0.98.62:45071)
0924 23:06:33.551467 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:33.551469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:33.551571 (+ 102us) server_negotiation.cc:407] Connection header received
0924 23:06:33.551649 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:33.551650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:33.551660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:33.551718 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0924 23:06:33.551872 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.552360 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.553485 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.553654 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.553703 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.553956 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 23:06:33.639118 (+ 85162us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 23:06:33.639280 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54674: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 23:06:33.639451 5871 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54674: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 23:06:33.639477 5896 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:33.551317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45071 (local address 127.0.0.1:54674)
0924 23:06:33.551505 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:33.551517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:33.551583 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:33.551743 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:33.551744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:33.551746 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 23:06:33.551807 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:33.551809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.552458 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.552460 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:33.553383 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:33.553386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:33.553741 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:33.553747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:33.553809 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:33.553815 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 23:06:33.639260 (+ 85445us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 23:06:33.639390 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45071: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260924 23:06:33.639935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5800
2026-09-24T23:06:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (307 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:33 dist-test-slave-ph9j krb5kdc[5908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:33 dist-test-slave-ph9j krb5kdc[5908](info): set up 2 sockets
Sep 24 23:06:33 dist-test-slave-ph9j krb5kdc[5908](info): commencing operation
krb5kdc: starting...
W20260924 23:06:34.846074 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:34 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:34.968340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:35.048182 5921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:35.048374 5921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:35.048396 5921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:35.049448 5921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:35.049482 5921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:35.049496 5921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:35.049508 5921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:35.049520 5921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:35.050935 5921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:35.051110 5921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:35.051260 5921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:35.055982 5926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.056005 5927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.055985 5929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:35.056180 5921 server_base.cc:1034] running on GCE node
I20260924 23:06:35.056464 5921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:35.056737 5921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:35.057933 5921 hybrid_clock.cc:648] HybridClock initialized: now 1790291195057909 us; error 40 us; skew 500 ppm
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.060241 5921 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:35.060658 5921 webserver.cc:460] Webserver started at http://127.0.98.62:35817/ using document root <none> and password file <none>
I20260924 23:06:35.060891 5921 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:35.060961 5921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:35.061110 5921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:35.062207 5921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "f9b52d095a11405db47de5d293d25dee"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "c90ce8caaea59dec2a946ded655e70e6"
server_key_iv: "b0850f997642c0009bdaae862409fc61"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.062604 5921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9b52d095a11405db47de5d293d25dee"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "c90ce8caaea59dec2a946ded655e70e6"
server_key_iv: "b0850f997642c0009bdaae862409fc61"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.063846 5921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:35.064508 5936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.064682 5921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:35.064769 5921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "f9b52d095a11405db47de5d293d25dee"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "c90ce8caaea59dec2a946ded655e70e6"
server_key_iv: "b0850f997642c0009bdaae862409fc61"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.064854 5921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:35.090417 5921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:35.092504 5921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:35.092653 5921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:35.096627 5921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38743
I20260924 23:06:35.096666 5988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38743 every 8 connection(s)
I20260924 23:06:35.097172 5921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:35.097959 5989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:35.100416 5989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee: Bootstrap starting.
I20260924 23:06:35.101003 5989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:35.101298 5989 log.cc:824] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:35.102046 5989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee: No bootstrap required, opened a new log
I20260924 23:06:35.102958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 5921
I20260924 23:06:35.103068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:35.103269 392 external_mini_cluster.cc:1428] Setting key e326c2e0848fb7c600be47c74f745acc
I20260924 23:06:35.104564 5989 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b52d095a11405db47de5d293d25dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260924 23:06:35.104728 5989 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:35.104794 5989 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b52d095a11405db47de5d293d25dee, State: Initialized, Role: FOLLOWER
I20260924 23:06:35.104971 5989 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b52d095a11405db47de5d293d25dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260924 23:06:35.105086 5989 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:35.105130 5989 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:35.105181 5989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:35.105723 5989 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b52d095a11405db47de5d293d25dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260924 23:06:35.105846 5989 leader_election.cc:302] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b52d095a11405db47de5d293d25dee; no voters:
I20260924 23:06:35.106055 5989 leader_election.cc:288] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:35.106086 5994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:35.106195 5994 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [term 1 LEADER]: Becoming Leader. State: Replica: f9b52d095a11405db47de5d293d25dee, State: Running, Role: LEADER
I20260924 23:06:35.106343 5989 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:35.106346 5994 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b52d095a11405db47de5d293d25dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260924 23:06:35.106801 5995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b52d095a11405db47de5d293d25dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b52d095a11405db47de5d293d25dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260924 23:06:35.107120 5995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:35.106985 5996 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b52d095a11405db47de5d293d25dee. Latest consensus state: current_term: 1 leader_uuid: "f9b52d095a11405db47de5d293d25dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b52d095a11405db47de5d293d25dee" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260924 23:06:35.107398 5996 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:35.107424 6003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:35.107983 6003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:35.110035 6003 catalog_manager.cc:1347] Generated new cluster ID: fc3dd0f250dc40d6942aaea6602bc38e
I20260924 23:06:35.110088 6003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:35.115386 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.103824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.98.62:38743)
0924 23:06:35.104012 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.104021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.104058 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:06:35.104557 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.104562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.104609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.104753 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.105477 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.106643 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.107661 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.108269 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.108368 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.111473 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.111477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.111478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.111491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.113649 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.114049 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.114053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.114053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.114180 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.114386 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.114387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.114388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.114707 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.114898 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.114978 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.115156 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260924 23:06:35.122695 6003 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:35.123456 6003 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:35.128872 6003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b52d095a11405db47de5d293d25dee: Generated new TSK 0
I20260924 23:06:35.129065 6003 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:35.154740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:35.236366 6015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:35.236575 6015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:35.236599 6015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:35.237752 6015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:35.237793 6015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:35.237820 6015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:35.239365 6015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:35.239614 6015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:35.239833 6015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:35.244867 6023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.244953 6021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.244884 6020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:35.244977 6015 server_base.cc:1034] running on GCE node
I20260924 23:06:35.245368 6015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:35.245615 6015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:35.246795 6015 hybrid_clock.cc:648] HybridClock initialized: now 1790291195246789 us; error 40 us; skew 500 ppm
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.249450 6015 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:35.249987 6015 webserver.cc:460] Webserver started at http://127.0.98.1:40941/ using document root <none> and password file <none>
I20260924 23:06:35.250211 6015 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:35.250281 6015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:35.250439 6015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:35.251454 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "47df65e4e345491ca2ff39fe611f52d3"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "728ee0f5adcd160917bafc612504d9f8"
server_key_iv: "981e1368edbcb953abc6f668bdba7d33"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.251853 6015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47df65e4e345491ca2ff39fe611f52d3"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "728ee0f5adcd160917bafc612504d9f8"
server_key_iv: "981e1368edbcb953abc6f668bdba7d33"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.253175 6015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:35.253861 6030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.254060 6015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:35.254142 6015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "47df65e4e345491ca2ff39fe611f52d3"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "728ee0f5adcd160917bafc612504d9f8"
server_key_iv: "981e1368edbcb953abc6f668bdba7d33"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.254194 6015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:35.266559 6015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:35.269297 6015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:35.269418 6015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:35.269624 6015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:35.270005 6015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:35.270044 6015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.270067 6015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:35.270080 6015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.275841 6015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45465
I20260924 23:06:35.275942 6143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45465 every 8 connection(s)
I20260924 23:06:35.276254 6015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:35.279467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6015
I20260924 23:06:35.279582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:35.279767 392 external_mini_cluster.cc:1428] Setting key 58a4cadf87e73c233d90d64b0f2ef3d2
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:35.286849 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.277076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40631 (local address 127.0.98.62:38743)
0924 23:06:35.277130 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.277132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.277365 (+ 233us) server_negotiation.cc:407] Connection header received
0924 23:06:35.278401 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.278402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.278412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.278481 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.279197 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.280018 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.281097 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.281312 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.281407 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.283684 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.283685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.283685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.283696 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.285317 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.285784 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.285786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.285786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.285848 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.286156 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.286157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.286157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.286282 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.286363 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.286663 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.286753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:35.287127 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.277036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:40631)
0924 23:06:35.277226 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:35.277268 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:35.278286 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:35.278625 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:35.278629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:35.278900 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:35.279107 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:35.279110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.280122 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.280123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:35.280982 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:35.280984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.281431 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.281432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:35.281565 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.282190 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:35.282200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:35.283572 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:35.285429 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:35.285438 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:35.285441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:35.285701 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:35.285966 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:35.285967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:35.285967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:35.286085 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:35.286353 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:35.286360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:35.286516 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 23:06:35.286626 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 23:06:35.286808 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260924 23:06:35.287706 6144 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260924 23:06:35.287827 6144 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:35.288130 6144 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 23:06:35.288805 5953 ts_manager.cc:194] Registered new tserver with Master: 47df65e4e345491ca2ff39fe611f52d3 (127.0.98.1:45465)
I20260924 23:06:35.289520 5953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:35.309476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:35.397688 6149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:35.397994 6149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:35.398067 6149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:35.399257 6149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:35.399340 6149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:35.399387 6149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:35.400962 6149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:35.401258 6149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:35.401511 6149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:35.406914 6155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.406966 6157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.406932 6154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:35.407274 6149 server_base.cc:1034] running on GCE node
I20260924 23:06:35.407553 6149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:35.407831 6149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:35.408985 6149 hybrid_clock.cc:648] HybridClock initialized: now 1790291195408954 us; error 49 us; skew 500 ppm
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.411695 6149 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:35.412143 6149 webserver.cc:460] Webserver started at http://127.0.98.2:42125/ using document root <none> and password file <none>
I20260924 23:06:35.412379 6149 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:35.412431 6149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:35.412563 6149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:35.413646 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "12576c175af34910ab6337a1921bd91d"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "724a46cb157f3ebfd337f86133699fdf"
server_key_iv: "0f2307aaa75dbf8ab90d50ea9c1ed01c"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.414034 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12576c175af34910ab6337a1921bd91d"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "724a46cb157f3ebfd337f86133699fdf"
server_key_iv: "0f2307aaa75dbf8ab90d50ea9c1ed01c"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.415479 6149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:35.416273 6164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.416464 6149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:35.416553 6149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "12576c175af34910ab6337a1921bd91d"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "724a46cb157f3ebfd337f86133699fdf"
server_key_iv: "0f2307aaa75dbf8ab90d50ea9c1ed01c"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.416605 6149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:35.448279 6149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:35.450784 6149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:35.450910 6149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:35.451103 6149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:35.451433 6149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:35.451469 6149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.451493 6149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:35.451541 6149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.457517 6149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37875
I20260924 23:06:35.457576 6277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37875 every 8 connection(s)
I20260924 23:06:35.457993 6149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:35.465169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6149
I20260924 23:06:35.465298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:35.465512 392 external_mini_cluster.cc:1428] Setting key 58606ce13f551495f91dd24b1943b5f5
I20260924 23:06:35.467974 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.458812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53015 (local address 127.0.98.62:38743)
0924 23:06:35.458924 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.458926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.459115 (+ 189us) server_negotiation.cc:407] Connection header received
0924 23:06:35.460103 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.460104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.460116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.460174 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.460816 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.461324 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.462397 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.462604 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.462688 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.464711 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.464712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.464713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.464719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.466446 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.466886 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.466888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.466888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.466953 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.467210 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.467211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.467212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.467334 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.467420 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.467778 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.467863 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 23:06:35.468192 6280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.458738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:53015)
0924 23:06:35.458983 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:35.459031 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:35.459982 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:35.460251 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:35.460256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:35.460525 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:35.460720 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:35.460723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.461415 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.461416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:35.462308 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:35.462310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.462697 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.462698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:35.462827 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.463308 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:35.463317 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:35.464610 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:35.466540 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:35.466547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:35.466550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:35.466785 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:35.467024 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:35.467024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:35.467025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:35.467141 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:35.467432 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:35.467439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:35.467623 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 23:06:35.467741 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 23:06:35.467957 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260924 23:06:35.468662 6278 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260924 23:06:35.468752 6278 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:35.468887 6278 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260924 23:06:35.469238 5953 ts_manager.cc:194] Registered new tserver with Master: 12576c175af34910ab6337a1921bd91d (127.0.98.2:37875)
I20260924 23:06:35.469769 5953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53015
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:35.493477 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:35.573728 6283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:35.574019 6283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:35.574062 6283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:35.575281 6283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:35.575356 6283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:35.575402 6283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:35.576864 6283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:35.577111 6283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:35.577332 6283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:35.582422 6288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:35.582448 6283 server_base.cc:1034] running on GCE node
W20260924 23:06:35.582444 6289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:35.582422 6291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:35.582878 6283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:35.583129 6283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:35.584355 6283 hybrid_clock.cc:648] HybridClock initialized: now 1790291195584340 us; error 30 us; skew 500 ppm
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.587328 6283 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:35.587754 6283 webserver.cc:460] Webserver started at http://127.0.98.3:36189/ using document root <none> and password file <none>
I20260924 23:06:35.587976 6283 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:35.588022 6283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:35.588119 6283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:35.589202 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "21acded57bd64ec0b7fb9ead3ea0a426"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "b23acdc62ee9dd62d03c0ef5397e7073"
server_key_iv: "5370665d6d59a90787c61b298cefa1f7"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.589550 6283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "21acded57bd64ec0b7fb9ead3ea0a426"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "b23acdc62ee9dd62d03c0ef5397e7073"
server_key_iv: "5370665d6d59a90787c61b298cefa1f7"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.590997 6283 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:35.591734 6298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.591922 6283 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:35.592028 6283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "21acded57bd64ec0b7fb9ead3ea0a426"
format_stamp: "Formatted at 2026-09-24 23:06:35 on dist-test-slave-ph9j"
server_key: "b23acdc62ee9dd62d03c0ef5397e7073"
server_key_iv: "5370665d6d59a90787c61b298cefa1f7"
server_key_version: "encryptionkey@0"
I20260924 23:06:35.592103 6283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:35.629733 6283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:35.632400 6283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:35.632586 6283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:35.632876 6283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:35.633210 6283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:35.633246 6283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.633313 6283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:35.633368 6283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:35.639596 6283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38171
I20260924 23:06:35.639642 6411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38171 every 8 connection(s)
I20260924 23:06:35.640003 6283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:35.648627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6283
I20260924 23:06:35.648725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:35.648929 392 external_mini_cluster.cc:1428] Setting key 9810e7ec04c3f748fa1624df13545a59
I20260924 23:06:35.649922 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.640792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47823 (local address 127.0.98.62:38743)
0924 23:06:35.640860 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.640862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.641114 (+ 252us) server_negotiation.cc:407] Connection header received
0924 23:06:35.642147 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.642148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.642159 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.642220 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.642888 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.643390 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.644422 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.644610 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.644671 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.646910 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.646912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.646912 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.646919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.648238 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.648819 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.648820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.648821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.648873 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.649165 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.649166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.649166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.649318 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.649386 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.649723 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.649803 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:06:35.650177 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.640767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:47823)
0924 23:06:35.640962 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:35.641026 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:35.642042 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:35.642298 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:35.642302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:35.642561 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:35.642754 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:35.642757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.643496 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.643497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:35.644331 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:35.644333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.644684 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.644685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:35.644788 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.645313 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:35.645321 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:35.646785 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:35.648327 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:35.648333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:35.648335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:35.648688 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:35.648947 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:35.648947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:35.648948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:35.649097 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:35.649389 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:35.649395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:35.649572 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 23:06:35.649691 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 23:06:35.649942 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260924 23:06:35.650671 6412 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260924 23:06:35.650764 6412 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:35.650902 6412 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260924 23:06:35.651266 5953 ts_manager.cc:194] Registered new tserver with Master: 21acded57bd64ec0b7fb9ead3ea0a426 (127.0.98.3:38171)
I20260924 23:06:35.651592 5953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47823
I20260924 23:06:35.660677 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/test-admin.keytab.
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.674960 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 23:06:35.676599 6419 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:35.675819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.0.98.1:45465)
0924 23:06:35.675995 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.676004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.676017 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:06:35.676042 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.676044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.676057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.676140 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.676513 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54336: BlockingRecv error: recv got EOF from 127.0.0.1:54336 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
W20260924 23:06:35.676720 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc)
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.677747 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.678714 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:35.686889 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.679258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.0.98.1:45465)
0924 23:06:35.679318 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.679320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.679394 (+ 74us) server_negotiation.cc:407] Connection header received
0924 23:06:35.679473 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.679474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.679486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.679560 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.679961 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.680685 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.681422 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.681754 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.681853 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.683993 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.684002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.684002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.684010 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.685642 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.686055 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.686058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.686059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.686119 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.686337 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.686338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.686338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.686475 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.686637 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.686691 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.686762 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 23:06:35.688704 6419 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:35.688078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.0.98.1:45465)
0924 23:06:35.688111 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.688113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.688151 (+ 38us) server_negotiation.cc:407] Connection header received
0924 23:06:35.688197 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.688198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.688206 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.688270 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.688645 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54352: BlockingRecv error: recv got EOF from 127.0.0.1:54352 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:06:35.688738 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.689657 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.690567 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:35.698405 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.691092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.0.98.1:45465)
0924 23:06:35.691145 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.691146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.691210 (+ 64us) server_negotiation.cc:407] Connection header received
0924 23:06:35.691297 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.691297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.691307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.691375 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.691731 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.692243 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.692959 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.693162 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.693248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.695496 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.695497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.695498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.695504 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.697006 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.697482 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.697484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.697484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.697549 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.697778 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.697779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.697780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.697940 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.698064 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.698185 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.698297 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 23:06:35.699788 6419 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:35.699364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.0.98.1:45465)
0924 23:06:35.699401 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.699402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.699411 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:35.699471 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.699471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.699479 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.699533 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.699751 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54366: BlockingRecv error: recv got EOF from 127.0.0.1:54366 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:06:35.699882 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790291169445806-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.700973 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:35.701835 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[5908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:06:35.708762 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:35.702417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.0.98.1:45465)
0924 23:06:35.702455 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:35.702462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:35.702492 (+ 30us) server_negotiation.cc:407] Connection header received
0924 23:06:35.702540 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:35.702540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:35.702547 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:35.702597 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:35.702929 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.703390 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.704121 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:35.704295 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:35.704353 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:35.706247 (+ 1894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:35.706248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:35.706248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:35.706253 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:35.707768 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.708049 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.708050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.708050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.708095 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:35.708288 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:35.708289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:35.708289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:35.708373 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:35.708442 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:35.708580 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 23:06:35.708632 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:06:35.709408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6015
I20260924 23:06:35.714254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6149
I20260924 23:06:35.718736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6283
I20260924 23:06:35.722679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 5921
2026-09-24T23:06:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1909 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[6432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[6432](info): set up 2 sockets
Sep 24 23:06:35 dist-test-slave-ph9j krb5kdc[6432](info): commencing operation
krb5kdc: starting...
W20260924 23:06:36.766165 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:36 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 23:06:36.886430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:36.969556 6445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:36.969800 6445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:36.969837 6445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:36.971096 6445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:36.971161 6445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:36.971189 6445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:36.971221 6445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:36.971251 6445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:36.972805 6445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:36.973090 6445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:36.973328 6445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:36.978374 6450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:36.978406 6451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:36.978633 6445 server_base.cc:1034] running on GCE node
W20260924 23:06:36.978374 6453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:36.978927 6445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:36.979179 6445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:36.980363 6445 hybrid_clock.cc:648] HybridClock initialized: now 1790291196980350 us; error 28 us; skew 500 ppm
Sep 24 23:06:36 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:36.983062 6445 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 23:06:36.983589 6445 webserver.cc:460] Webserver started at http://127.0.98.62:45337/ using document root <none> and password file <none>
I20260924 23:06:36.983824 6445 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:36.983892 6445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:36.984054 6445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:36.985158 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a"
format_stamp: "Formatted at 2026-09-24 23:06:36 on dist-test-slave-ph9j"
server_key: "83f11b73a8f2251e484b2f1848352abf"
server_key_iv: "e4ca37aab279da2b0eafafcf646da77b"
server_key_version: "encryptionkey@0"
I20260924 23:06:36.985535 6445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a"
format_stamp: "Formatted at 2026-09-24 23:06:36 on dist-test-slave-ph9j"
server_key: "83f11b73a8f2251e484b2f1848352abf"
server_key_iv: "e4ca37aab279da2b0eafafcf646da77b"
server_key_version: "encryptionkey@0"
I20260924 23:06:36.987032 6445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:36.987896 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:36.988111 6445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:36.988229 6445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a"
format_stamp: "Formatted at 2026-09-24 23:06:36 on dist-test-slave-ph9j"
server_key: "83f11b73a8f2251e484b2f1848352abf"
server_key_iv: "e4ca37aab279da2b0eafafcf646da77b"
server_key_version: "encryptionkey@0"
I20260924 23:06:36.988318 6445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.005514 6445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.007977 6445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:37.008162 6445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.012494 6445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38659
I20260924 23:06:37.012528 6512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38659 every 8 connection(s)
I20260924 23:06:37.012907 6445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:37.013947 6513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:37.016280 6513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Bootstrap starting.
I20260924 23:06:37.016881 6513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:37.017171 6513 log.cc:824] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:37.017959 6513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: No bootstrap required, opened a new log
I20260924 23:06:37.019508 6513 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260924 23:06:37.019655 6513 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:37.019678 6513 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c2dd1a93dd4b0eb6e0ef7991a4ec6a, State: Initialized, Role: FOLLOWER
I20260924 23:06:37.019834 6513 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260924 23:06:37.019939 6513 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:37.019984 6513 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:37.020058 6513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:37.020879 6513 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260924 23:06:37.020994 6513 leader_election.cc:302] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3c2dd1a93dd4b0eb6e0ef7991a4ec6a; no voters:
I20260924 23:06:37.021166 6513 leader_election.cc:288] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:37.021204 6516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:37.021436 6513 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:37.021462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6445
I20260924 23:06:37.021476 6516 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 LEADER]: Becoming Leader. State: Replica: d3c2dd1a93dd4b0eb6e0ef7991a4ec6a, State: Running, Role: LEADER
I20260924 23:06:37.021564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:37.021719 6516 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260924 23:06:37.021822 392 external_mini_cluster.cc:1428] Setting key a9db315982d80f3462610532621f0095
I20260924 23:06:37.022176 6517 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } }
I20260924 23:06:37.022253 6517 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:37.022692 6526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:37.022677 6518 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } }
I20260924 23:06:37.022960 6518 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:37.023240 6526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:37.026954 6526 catalog_manager.cc:1347] Generated new cluster ID: 90ab479602ae482a96c9c1a70ffee77a
I20260924 23:06:37.027032 6526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:37.034314 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.022536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.0.98.62:38659)
0924 23:06:37.023196 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.023206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.023514 (+ 308us) server_negotiation.cc:407] Connection header received
0924 23:06:37.023629 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.023635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.023698 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.023831 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.024534 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.026010 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.026929 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.027582 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.027696 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.030627 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.030632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.030632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.030645 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.032605 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.033031 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.033033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.033034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.033150 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.033363 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.033364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.033364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.033664 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.033820 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.033950 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.034088 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":80,"threads_started":1}
I20260924 23:06:37.036731 6526 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:37.037202 6526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:37.045342 6526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Generated new TSK 0
I20260924 23:06:37.045524 6526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 23:06:37.076704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:37.158429 6539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:37.158666 6539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:37.158704 6539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:37.159874 6539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:37.159947 6539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:37.159993 6539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:37.161453 6539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:37.161722 6539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:37.161988 6539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:37.166980 6545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.166988 6544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.167021 6539 server_base.cc:1034] running on GCE node
W20260924 23:06:37.166988 6547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.167387 6539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:37.167593 6539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:37.168766 6539 hybrid_clock.cc:648] HybridClock initialized: now 1790291197168755 us; error 38 us; skew 500 ppm
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:37.171494 6539 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 23:06:37.172003 6539 webserver.cc:460] Webserver started at http://127.0.98.1:42589/ using document root <none> and password file <none>
I20260924 23:06:37.172232 6539 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.172302 6539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.172461 6539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:37.173550 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b487ca706f14b5dabeb532009e2b2ed"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "2e2f46d1c30bf23b713a3706062be49f"
server_key_iv: "4829be74e26123cef39e53efc38d67d3"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.173985 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b487ca706f14b5dabeb532009e2b2ed"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "2e2f46d1c30bf23b713a3706062be49f"
server_key_iv: "4829be74e26123cef39e53efc38d67d3"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.175338 6539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:37.176148 6554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.176340 6539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:37.176442 6539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "4b487ca706f14b5dabeb532009e2b2ed"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "2e2f46d1c30bf23b713a3706062be49f"
server_key_iv: "4829be74e26123cef39e53efc38d67d3"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.176529 6539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.188799 6539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.190908 6539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:37.191107 6539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.191365 6539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:37.191730 6539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:37.191785 6539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.191828 6539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:37.191890 6539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.197748 6539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37045
I20260924 23:06:37.197790 6667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37045 every 8 connection(s)
I20260924 23:06:37.198191 6539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:37.201824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6539
I20260924 23:06:37.201973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:37.202179 392 external_mini_cluster.cc:1428] Setting key 04056cfbe921d8115b101d2c2c01ceb5
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:37.208632 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.199000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50075 (local address 127.0.98.62:38659)
0924 23:06:37.199053 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.199057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.199251 (+ 194us) server_negotiation.cc:407] Connection header received
0924 23:06:37.199735 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.199737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.199752 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.199828 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.200770 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.201328 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.202733 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.203015 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.203071 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.205500 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.205502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.205502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.205511 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.207071 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.207586 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.207587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.207588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.207651 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.207970 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.207971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.207972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.208096 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.208169 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.208453 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.208541 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 23:06:37.208853 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.198918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:50075)
0924 23:06:37.199075 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.199118 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.199615 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.199890 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.199893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.200473 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.200667 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.200672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.201413 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.201413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.202648 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.202651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.203107 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.203108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.203259 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.203805 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.203817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.205398 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.207177 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.207184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.207187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.207481 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.207732 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.207733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.207733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.207895 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.208171 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.208178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.208318 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.208432 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.208595 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260924 23:06:37.209508 6668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260924 23:06:37.209625 6668 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:37.209843 6668 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 23:06:37.210532 6477 ts_manager.cc:194] Registered new tserver with Master: 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045)
I20260924 23:06:37.211164 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 23:06:37.228488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:37.317683 6673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:37.317934 6673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:37.317981 6673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:37.319144 6673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:37.319187 6673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:37.319243 6673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:37.322125 6673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:37.322393 6673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:37.322683 6673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:37.327710 6678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.327699 6681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.327702 6679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.328683 6673 server_base.cc:1034] running on GCE node
I20260924 23:06:37.328912 6673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:37.329159 6673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:37.330338 6673 hybrid_clock.cc:648] HybridClock initialized: now 1790291197330325 us; error 31 us; skew 500 ppm
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:37.332799 6673 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 23:06:37.333238 6673 webserver.cc:460] Webserver started at http://127.0.98.2:37883/ using document root <none> and password file <none>
I20260924 23:06:37.333460 6673 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.333504 6673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.333665 6673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:37.334918 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "61f6d6984b964a15a5498ce99dfd41f5"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "106fd69b8c579d65263328649abd4912"
server_key_iv: "4a169cc3a146a2a81937db46e9d2e0c3"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.335276 6673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "61f6d6984b964a15a5498ce99dfd41f5"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "106fd69b8c579d65263328649abd4912"
server_key_iv: "4a169cc3a146a2a81937db46e9d2e0c3"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.336694 6673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:37.337524 6688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.337718 6673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:37.337849 6673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "61f6d6984b964a15a5498ce99dfd41f5"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "106fd69b8c579d65263328649abd4912"
server_key_iv: "4a169cc3a146a2a81937db46e9d2e0c3"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.337961 6673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.344375 6673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.346462 6673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:37.346572 6673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.346740 6673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:37.347082 6673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:37.347115 6673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.347137 6673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:37.347172 6673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.352679 6673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37683
I20260924 23:06:37.352751 6801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37683 every 8 connection(s)
I20260924 23:06:37.353073 6673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:37.353372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6673
I20260924 23:06:37.353475 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:37.353682 392 external_mini_cluster.cc:1428] Setting key 3a45fcb1a67db74f0c19024eb0976338
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 23:06:37.363605 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.353933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33527 (local address 127.0.98.62:38659)
0924 23:06:37.354006 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.354009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.354423 (+ 414us) server_negotiation.cc:407] Connection header received
0924 23:06:37.354946 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.354947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.354959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.355018 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.355940 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.356467 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.357475 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.357831 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.357931 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.360362 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.360363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.360363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.360369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.361770 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.362385 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.362386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.362387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.362438 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.362887 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.362888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.362888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.362995 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.363049 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.363431 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.363514 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:06:37.363710 6804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.353853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:33527)
0924 23:06:37.354291 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.354351 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.354836 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.355085 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.355088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.355646 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.355835 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.355839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.356557 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.356558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.357353 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.357379 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.357953 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.357954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.358067 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.358579 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.358588 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.360271 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.361871 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.361929 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.361932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.362158 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.362644 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.362645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.362646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.362811 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.363056 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.363061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.363179 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.363268 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.363406 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260924 23:06:37.364332 6802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260924 23:06:37.364463 6802 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:37.364684 6802 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260924 23:06:37.365121 6477 ts_manager.cc:194] Registered new tserver with Master: 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683)
I20260924 23:06:37.365494 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 23:06:37.380877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:37.460983 6807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:37.461205 6807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:37.461230 6807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:37.462401 6807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:37.462451 6807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:37.462479 6807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:37.463966 6807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:37.464170 6807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:37.464358 6807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:37.469650 6815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.469656 6812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.469661 6813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.470096 6807 server_base.cc:1034] running on GCE node
I20260924 23:06:37.470285 6807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:37.470531 6807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:37.471724 6807 hybrid_clock.cc:648] HybridClock initialized: now 1790291197471700 us; error 33 us; skew 500 ppm
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:37.474555 6807 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 23:06:37.475060 6807 webserver.cc:460] Webserver started at http://127.0.98.3:41245/ using document root <none> and password file <none>
I20260924 23:06:37.475265 6807 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.475309 6807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.475401 6807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:37.476435 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ae6392fb8a648d59312a7b169c22d0b"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "0906ad9fc4a9f324658fef7b8d588e7d"
server_key_iv: "f394a16584508c90b598b5be26d805bf"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.476747 6807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ae6392fb8a648d59312a7b169c22d0b"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "0906ad9fc4a9f324658fef7b8d588e7d"
server_key_iv: "f394a16584508c90b598b5be26d805bf"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.478219 6807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:06:37.479116 6822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.479441 6807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:37.479573 6807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "4ae6392fb8a648d59312a7b169c22d0b"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "0906ad9fc4a9f324658fef7b8d588e7d"
server_key_iv: "f394a16584508c90b598b5be26d805bf"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.479661 6807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.492399 6807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.494879 6807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:37.495031 6807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.495291 6807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:37.495622 6807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:37.495656 6807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.495678 6807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:37.495720 6807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.501961 6807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46601
I20260924 23:06:37.502045 6935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46601 every 8 connection(s)
I20260924 23:06:37.502370 6807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:37.505738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6807
I20260924 23:06:37.505825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:37.506033 392 external_mini_cluster.cc:1428] Setting key 232c87b5ee83d90e4fa5c551a772a457
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:37.512128 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.503145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55141 (local address 127.0.98.62:38659)
0924 23:06:37.503240 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.503242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.503429 (+ 187us) server_negotiation.cc:407] Connection header received
0924 23:06:37.503910 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.503911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.503921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.503985 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.504966 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.505458 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.506885 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.507147 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.507203 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.509422 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.509424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.509424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.509431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.510948 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.511337 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.511338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.511339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.511391 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.511607 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.511608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.511608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.511715 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.511766 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.512002 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.512062 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 23:06:37.512297 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.503123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:55141)
0924 23:06:37.503297 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.503340 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.503819 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.504079 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.504083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.504656 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.504857 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.504861 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.505542 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.505543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.506790 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.506793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.507220 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.507221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.507325 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.507805 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.507813 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.509334 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.511019 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.511024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.511025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.511258 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.511452 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.511452 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.511453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.511548 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.511770 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.511774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.511893 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.511973 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.512096 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260924 23:06:37.512761 6936 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260924 23:06:37.512853 6936 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:37.513078 6936 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260924 23:06:37.513463 6477 ts_manager.cc:194] Registered new tserver with Master: 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601)
I20260924 23:06:37.513808 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55141
I20260924 23:06:37.517746 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 23:06:37.540994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40649
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38659,127.0.98.61:40649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:37.622319 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:37.622536 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:37.622560 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:37.623744 6941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:37.623787 6941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:37.623804 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:37.623817 6941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:37.623829 6941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:37.625337 6941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38659,127.0.98.61:40649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
W20260924 23:06:37.625468 6941 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38659,127.0.98.61:40649'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 23:06:37.625617 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:37.625855 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:06:37.630946 6941 server_base.cc:1034] running on GCE node
W20260924 23:06:37.630966 6946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.630862 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.630895 6947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.631464 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:37.631791 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:37.632948 6941 hybrid_clock.cc:648] HybridClock initialized: now 1790291197632933 us; error 30 us; skew 500 ppm
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:37.635481 6941 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 23:06:37.635958 6941 webserver.cc:460] Webserver started at http://127.0.98.61:34301/ using document root <none> and password file <none>
I20260924 23:06:37.636147 6941 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.636186 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.636277 6941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:37.637305 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/instance:
uuid: "6386922bf36145bf863f98aeb7dd9de3"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "18fb343793bab8e11cc6e4761b445949"
server_key_iv: "ef58afebf60523ce3147746863c9da59"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.637633 6941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal/instance:
uuid: "6386922bf36145bf863f98aeb7dd9de3"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "18fb343793bab8e11cc6e4761b445949"
server_key_iv: "ef58afebf60523ce3147746863c9da59"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.639038 6941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:37.639843 6956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.640079 6941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:37.640182 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
uuid: "6386922bf36145bf863f98aeb7dd9de3"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "18fb343793bab8e11cc6e4761b445949"
server_key_iv: "ef58afebf60523ce3147746863c9da59"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.640264 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.647883 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.649716 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:37.649820 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.654017 6941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40649
I20260924 23:06:37.654106 7008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40649 every 8 connection(s)
I20260924 23:06:37.654528 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
I20260924 23:06:37.655495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 6941
I20260924 23:06:37.655524 7009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:37.658046 7009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:37.666414 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.656142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.0.98.61:40649)
0924 23:06:37.656314 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.656323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.656357 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:06:37.656967 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.656973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.657021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.657135 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.658152 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.659300 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.660054 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.660454 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.660579 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.662980 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.662988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.662988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.663001 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.664817 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.665207 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.665210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.665210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.665299 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.665517 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.665518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.665518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.665748 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.665920 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.666055 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.666178 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260924 23:06:37.667189 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.658873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.0.98.62:38659)
0924 23:06:37.658941 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.658943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.659090 (+ 147us) server_negotiation.cc:407] Connection header received
0924 23:06:37.659203 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.659204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.659217 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.659305 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.659789 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.660320 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.661393 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.661607 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.661706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.663999 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.664001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.664002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.664011 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.665414 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.666067 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.666068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.666069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.666116 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.666360 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.666361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.666362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.666716 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.666832 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.667027 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.667102 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:37.667204 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.658743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43436)
0924 23:06:37.658989 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.659024 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.659120 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.659348 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.659351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.659616 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.659683 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.659685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.660431 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.660432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.661280 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.661283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.661735 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.661736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.661814 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.662412 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.662420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.663899 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.665547 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.665553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.665555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.665747 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.666193 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.666194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.666195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.666270 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.666836 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.666843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.666894 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.666987 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.667102 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260924 23:06:37.667819 7009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 23:06:37.673630 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.668146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.0.98.61:40649)
0924 23:06:37.668192 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.668193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.668207 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:06:37.668222 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.668223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.668230 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.668276 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.668497 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.668940 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.669527 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.669710 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.669760 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.671428 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.671429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.671429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.671434 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.672757 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.673073 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.673074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.673074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.673121 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.673266 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.673266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.673266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.673370 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.673415 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.673518 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.673573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:06:37.673964 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.668053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36340)
0924 23:06:37.668102 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.668113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.668156 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.668291 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.668292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.668399 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.668441 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.668443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.669001 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.669002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.669464 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.669466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.669774 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.669775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.669841 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.670199 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.670203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.671357 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.672854 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.672855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.672856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.673008 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.673180 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.673180 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.673181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.673228 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.673421 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.673421 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.673454 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.673503 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.673845 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 23:06:37.675458 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Bootstrap starting.
I20260924 23:06:37.676321 7009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:37.676602 7009 log.cc:824] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:37.677510 7009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: No bootstrap required, opened a new log
I20260924 23:06:37.679178 7009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } }
I20260924 23:06:37.679307 7009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:37.679330 7009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6386922bf36145bf863f98aeb7dd9de3, State: Initialized, Role: FOLLOWER
I20260924 23:06:37.679476 7009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } }
I20260924 23:06:37.679814 7009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:37.679747 7015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } } }
I20260924 23:06:37.679872 7015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:06:37.681641 7026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:06:37.681711 7026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:06:37.683272 6941 master_runner.cc:428] Detected that this master 6386922bf36145bf863f98aeb7dd9de3 is joining an existing cluster
I20260924 23:06:37.683320 6941 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40649
I20260924 23:06:37.691838 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.684653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.0.98.61:40649)
0924 23:06:37.684685 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.684687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.684865 (+ 178us) server_negotiation.cc:407] Connection header received
0924 23:06:37.684931 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.684932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.684940 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.684992 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.685397 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.686138 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.687081 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.687285 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.687374 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.688826 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.688827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.688827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.688834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.690696 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.691041 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.691042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.691043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.691093 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.691289 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.691289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.691290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.691407 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.691486 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.691701 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.691769 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:06:37.692058 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.684494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.0.98.62:38659)
0924 23:06:37.684584 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.684587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.684601 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:06:37.684629 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.684631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.684645 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.684718 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.685177 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.685759 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.686957 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.687266 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.687352 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.689097 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.689098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.689098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.689106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.690836 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.691226 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.691228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.691228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.691296 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.691500 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.691501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.691501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.691635 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.691674 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.691896 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.691975 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 23:06:37.692078 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.684680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36346)
0924 23:06:37.684816 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.684825 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.684878 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.684998 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.684998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.685241 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.685312 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.685314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.686223 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.686224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.687009 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.687010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.687418 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.687419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.687472 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.688065 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.688070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.688729 (+ 659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.690786 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.690787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.690787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.690958 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.691166 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.691166 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.691167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.691219 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.691500 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.691501 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.691542 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.691929 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.692021 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":102,"threads_started":1}
I20260924 23:06:37.692617 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.684337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43448)
0924 23:06:37.684468 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.684480 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.684534 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.684777 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.684779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.684991 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.685074 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.685075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.685855 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.685855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.686863 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.686865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.687369 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.687370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.687424 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.688134 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.688139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.688729 (+ 590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.690930 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.690930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.690931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.691141 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.691372 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.691372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.691373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.691428 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.691720 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.691721 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.691763 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.692409 (+ 646us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.692510 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 23:06:37.702301 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.694837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36356 (local address 127.0.98.61:40649)
0924 23:06:37.694873 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.694875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.695027 (+ 152us) server_negotiation.cc:407] Connection header received
0924 23:06:37.695118 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.695119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.695129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.695184 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.695603 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.696052 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.696836 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.697031 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.697107 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.699388 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.699390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.699390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.699396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.700987 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.701431 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.701432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.701432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.701485 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.701711 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.701712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.701713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.701860 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.701994 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.702171 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.702228 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:06:37.702371 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.694776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36356)
0924 23:06:37.694936 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.694966 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.695052 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.695190 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.695193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.695467 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.695514 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.695516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.696130 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.696131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.696726 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.696728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.697105 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.697106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.697194 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.697660 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.697665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.699297 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.701073 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.701080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.701082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.701346 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.701557 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.701557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.701558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.701624 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.701986 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.701993 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.702048 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.702140 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.702255 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260924 23:06:37.703167 6477 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40649: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40649
}
attrs {
promote: true
}
}
dest_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a"
cas_config_opid_index: -1
I20260924 23:06:37.703519 6477 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } }
I20260924 23:06:37.704027 7036 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6386922bf36145bf863f98aeb7dd9de3 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } }
I20260924 23:06:37.704373 6983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:37.705049 7036 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40649
I20260924 23:06:37.705230 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } }
I20260924 23:06:37.705284 6941 master.cc:537] Master@127.0.98.61:40649 shutting down...
I20260924 23:06:37.705318 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:37.705323 6983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Refusing update from remote peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 23:06:37.705705 7035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 23:06:37.706627 6464 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 23:06:37.707185 6464 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a -> Peer 6386922bf36145bf863f98aeb7dd9de3 (127.0.98.61:40649): Couldn't send request to peer 6386922bf36145bf863f98aeb7dd9de3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 23:06:37.708400 6941 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:37.708498 6941 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:37.708554 6941 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: stopping tablet replica
I20260924 23:06:37.721583 6941 master.cc:552] Master@127.0.98.61:40649 shutdown complete.
I20260924 23:06:37.721653 6941 master_runner.cc:305] Clearing existing system tablet
I20260924 23:06:37.721833 6941 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.721925 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.722688 6941 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:37.723217 7040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.723356 6941 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.723426 6941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:37.723454 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
uuid: "6386922bf36145bf863f98aeb7dd9de3"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "18fb343793bab8e11cc6e4761b445949"
server_key_iv: "ef58afebf60523ce3147746863c9da59"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.724193 6941 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:37.725764 6941 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 23:06:37.725833 6941 log.cc:1196] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 23:06:37.726042 6941 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Deleting consensus metadata
I20260924 23:06:37.726162 6941 master_runner.cc:315] Copying system tablet from 127.0.98.62:38659
I20260924 23:06:37.726763 6941 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38659
I20260924 23:06:37.733393 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.727058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43450)
0924 23:06:37.727232 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.727242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.727302 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.727437 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.727437 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.727596 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.727653 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.727654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.728603 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.728604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.729205 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.729207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.729681 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.729682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.729735 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.730424 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.730429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.730745 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.732478 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.732478 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.732479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.732645 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.732862 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.732863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.732863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.732914 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.733162 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.733163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.733197 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.733247 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.733321 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260924 23:06:37.733443 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.727121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.0.98.62:38659)
0924 23:06:37.727175 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.727178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.727313 (+ 135us) server_negotiation.cc:407] Connection header received
0924 23:06:37.727345 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.727345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.727357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.727434 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.727723 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.728506 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.729295 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.729601 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.729672 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.730837 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.730838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.730839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.730846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.732399 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.732715 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.732717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.732717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.732788 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.732974 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.732975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.732975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.733101 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.733169 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.733271 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.733351 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:37.734112 6497 tablet_copy_service.cc:140] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6386922bf36145bf863f98aeb7dd9de3 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43450)
I20260924 23:06:37.734215 6497 tablet_copy_service.cc:161] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6386922bf36145bf863f98aeb7dd9de3 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43450: session id = 6386922bf36145bf863f98aeb7dd9de3-00000000000000000000000000000000
I20260924 23:06:37.735131 6497 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 23:06:37.735803 6941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 23:06:37.737030 6941 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: tablet copy: Starting download of 0 data blocks...
I20260924 23:06:37.737180 6941 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: tablet copy: Starting download of 1 WAL segments...
I20260924 23:06:37.738111 6941 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 23:06:37.739015 6497 tablet_copy_service.cc:342] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Request end of tablet copy session 6386922bf36145bf863f98aeb7dd9de3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43450
I20260924 23:06:37.739090 6497 tablet_copy_service.cc:433] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: ending tablet copy session 6386922bf36145bf863f98aeb7dd9de3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6386922bf36145bf863f98aeb7dd9de3
I20260924 23:06:37.741447 6941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:37.742678 7047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.742731 7050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.742898 7048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.742974 6941 server_base.cc:1034] running on GCE node
I20260924 23:06:37.743130 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:37.743261 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:37.744393 6941 hybrid_clock.cc:648] HybridClock initialized: now 1790291197744384 us; error 25 us; skew 500 ppm
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:37.745685 6941 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 23:06:37.746014 6941 webserver.cc:460] Webserver started at http://127.0.98.61:43793/ using document root <none> and password file <none>
I20260924 23:06:37.746132 6941 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.746186 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.746789 6941 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:37.747337 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.747495 6941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:37.747552 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
uuid: "6386922bf36145bf863f98aeb7dd9de3"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "18fb343793bab8e11cc6e4761b445949"
server_key_iv: "ef58afebf60523ce3147746863c9da59"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.747611 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.763536 6941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.765388 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.768837 6941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40649
I20260924 23:06:37.769109 7109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40649 every 8 connection(s)
I20260924 23:06:37.769222 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
I20260924 23:06:37.770145 7110 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:06:37.770689 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Bootstrap starting.
I20260924 23:06:37.773694 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:06:37.773984 7110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Bootstrap complete.
I20260924 23:06:37.774196 7110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } }
I20260924 23:06:37.774300 7110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6386922bf36145bf863f98aeb7dd9de3, State: Initialized, Role: LEARNER
I20260924 23:06:37.774344 7110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } }
I20260924 23:06:37.774520 7110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:37.774544 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } }
I20260924 23:06:37.774684 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: LEARNER
I20260924 23:06:37.776051 7124 catalog_manager.cc:1259] Loaded cluster ID: 90ab479602ae482a96c9c1a70ffee77a
I20260924 23:06:37.776332 7124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: loading cluster ID for follower catalog manager: success
I20260924 23:06:37.777166 7124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: acquiring CA information for follower catalog manager: success
I20260924 23:06:37.777479 7124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:06:37.825960 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.818603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.0.98.61:40649)
0924 23:06:37.818770 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.818773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.818782 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:37.818812 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.818813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.818827 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.818915 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.819568 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.820278 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.821113 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.821368 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.821453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.823039 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.823041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.823041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.823049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.824615 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.825023 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.825024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.825025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.825092 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.825330 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.825331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.825331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.825459 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.825576 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.825704 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.825805 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 23:06:37.853127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39849
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:37.936508 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:37.936708 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:37.936729 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:37.937844 7128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:37.937924 7128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:37.937954 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:37.937968 7128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:37.938009 7128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:37.939433 7128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:37.939680 7128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:37.939906 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:37.944852 7136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.944867 7133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:37.944886 7134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:37.945061 7128 server_base.cc:1034] running on GCE node
I20260924 23:06:37.945248 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:37.945478 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:37.946615 7128 hybrid_clock.cc:648] HybridClock initialized: now 1790291197946605 us; error 27 us; skew 500 ppm
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:37.949352 7128 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 23:06:37.949704 7128 webserver.cc:460] Webserver started at http://127.0.98.60:46749/ using document root <none> and password file <none>
I20260924 23:06:37.949957 7128 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:37.950008 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:37.950095 7128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:37.951066 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/instance:
uuid: "0b5b0ff144db47fcb0b5996a9537555d"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "15378d1af74f9511cffd91954dc42f35"
server_key_iv: "9538768c2856d41e9a628eb55a2c2dad"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.951377 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal/instance:
uuid: "0b5b0ff144db47fcb0b5996a9537555d"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "15378d1af74f9511cffd91954dc42f35"
server_key_iv: "9538768c2856d41e9a628eb55a2c2dad"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.952638 7128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:37.953455 7143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:37.953697 7128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:37.953801 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
uuid: "0b5b0ff144db47fcb0b5996a9537555d"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "15378d1af74f9511cffd91954dc42f35"
server_key_iv: "9538768c2856d41e9a628eb55a2c2dad"
server_key_version: "encryptionkey@0"
I20260924 23:06:37.953883 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:37.971174 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:37.973080 7128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:37.973245 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:37.978240 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39849
I20260924 23:06:37.978269 7195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39849 every 8 connection(s)
I20260924 23:06:37.978699 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
I20260924 23:06:37.979868 7196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:37.981370 7196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:37.988169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 7128
I20260924 23:06:37.990348 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.982231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.0.98.62:38659)
0924 23:06:37.982292 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.982294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.982349 (+ 55us) server_negotiation.cc:407] Connection header received
0924 23:06:37.982809 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.982810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.982821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.982889 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.983492 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.984001 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.985000 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.985186 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.985245 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.987417 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.987419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.987419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.987425 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.988976 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.989393 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.989394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.989395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.989462 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.989706 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.989707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.989707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.989839 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.989937 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.990219 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.990278 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:06:37.990586 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.982082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43460)
0924 23:06:37.982230 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.982271 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.982715 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.982960 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.982964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.983213 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.983403 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.983406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.984092 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.984093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.984912 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.984914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.985279 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.985280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.985398 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.985958 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.985966 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.987330 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.989047 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.989052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.989054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.989312 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.989532 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.989533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.989533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.989635 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.989931 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.989936 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.990083 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.990196 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.990351 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 23:06:37.991212 7196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 23:06:37 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 23:06:37.998382 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.988724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.0.98.60:39849)
0924 23:06:37.988907 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.988913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.988949 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:06:37.988976 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.988978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.988989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.989071 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.989708 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.990765 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.991760 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.992236 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.992329 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.995244 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.995250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.995250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.995256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.996995 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.997430 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.997433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.997434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.997515 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.997751 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.997752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.997752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.998004 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.998127 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.998247 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.998311 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260924 23:06:37.999434 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.991459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36388)
0924 23:06:37.991510 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:37.991526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:37.991565 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:37.991997 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:37.991998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:37.992104 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:37.992146 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.992147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.993034 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.993035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.993979 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:37.993981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.994469 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.994469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:37.994525 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.994923 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:37.994929 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:37.996440 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:37.998437 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.998438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.998439 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.998634 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.998837 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:37.998838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:37.998839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:37.998895 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:37.999211 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:37.999211 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:37.999246 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 23:06:37.999306 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:06:37.999376 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:06:37.999449 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:37.991790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.0.98.61:40649)
0924 23:06:37.991874 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:37.991876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:37.991883 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:37.991905 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:37.991906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:37.991920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:37.991975 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:37.992222 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.992934 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.994065 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:37.994381 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:37.994462 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:37.996527 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:37.996529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:37.996529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:37.996535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:37.998355 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.998701 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.998702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.998703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.998763 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:37.998990 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:37.998992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:37.998992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:37.999147 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:37.999202 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:37.999323 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 23:06:37.999388 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 23:06:37.999874 7196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291197, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 23:06:38.005901 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.000115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60882)
0924 23:06:38.000164 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.000175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.000221 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.000342 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.000343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.000471 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.000496 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.000497 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.001119 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.001119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.001591 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.001592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.002008 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.002009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.002059 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.002386 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.002390 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.003510 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.004895 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.004896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.004896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.005054 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.005266 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.005267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.005267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.005332 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.005561 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.005562 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.005597 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.005667 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.005758 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 23:06:38.005950 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.000168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60882 (local address 127.0.98.60:39849)
0924 23:06:38.000195 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.000197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.000208 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:38.000275 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.000276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.000285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.000339 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.000566 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.001032 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.001680 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.001943 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.002009 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.003604 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.003605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.003606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.003612 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.004757 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.005136 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.005136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.005137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.005191 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.005391 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.005392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.005392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.005496 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.005570 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.005687 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.005812 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":14}
I20260924 23:06:38.007210 7196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Bootstrap starting.
I20260924 23:06:38.007978 7196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:38.008628 7196 log.cc:824] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:38.009433 7196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: No bootstrap required, opened a new log
I20260924 23:06:38.011088 7196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } }
I20260924 23:06:38.011246 7196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:38.011304 7196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5b0ff144db47fcb0b5996a9537555d, State: Initialized, Role: FOLLOWER
I20260924 23:06:38.011435 7196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } }
I20260924 23:06:38.011729 7196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:38.011746 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } } }
I20260924 23:06:38.011831 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:06:38.013507 7212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:06:38.013566 7212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:06:38.016590 7128 master_runner.cc:428] Detected that this master 0b5b0ff144db47fcb0b5996a9537555d is joining an existing cluster
I20260924 23:06:38.016662 7128 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39849
I20260924 23:06:38.041651 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.017730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.98.60:39849)
0924 23:06:38.017810 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.017812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.018039 (+ 227us) server_negotiation.cc:407] Connection header received
0924 23:06:38.018117 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.018117 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.018124 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.018149 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.018582 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.019068 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020087 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.020354 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020390 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.021914 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.021915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.021915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.021921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.040260 (+ 18339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.040654 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.040655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.040655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.040718 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.041046 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.041047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.041048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.041206 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.041251 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.041471 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.041550 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 23:06:38.041867 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.017850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.98.61:40649)
0924 23:06:38.017908 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.017910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.018058 (+ 148us) server_negotiation.cc:407] Connection header received
0924 23:06:38.018168 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.018169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.018187 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.018217 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.019630 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.020356 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.021411 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.021719 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.021797 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.038693 (+ 16896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.038695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.038695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.038704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.040506 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.041040 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.041041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.041042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.041100 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.041298 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.041298 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.041299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.041426 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.041466 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.041672 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.041759 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:06:38.042233 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.017858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36392)
0924 23:06:38.017996 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.018006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.018083 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.018282 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.018283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.019464 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.019537 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.019539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.020455 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.021323 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.021325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.021805 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.021806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.021872 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.022638 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.022642 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.038547 (+ 15905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.040635 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.040636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.040637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.040955 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.041167 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.041168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.041168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.041219 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.041511 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.041512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.041550 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.042024 (+ 474us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.042132 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260924 23:06:38.042481 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.017582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60886)
0924 23:06:38.017986 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.017996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.018058 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.018195 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.018196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.018449 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.018503 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.018505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.019146 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.019147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.020002 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.020003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.020466 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.020529 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.021359 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.021363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.021798 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.040341 (+ 18543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.040343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.040343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.040560 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.040838 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.040839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.040840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.040959 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.041301 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.041302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.041354 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.042262 (+ 908us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.042364 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":71,"threads_started":1}
I20260924 23:06:38.043620 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.017588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.0.98.62:38659)
0924 23:06:38.017642 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.017645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.017762 (+ 117us) server_negotiation.cc:407] Connection header received
0924 23:06:38.017916 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.017917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.017930 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.017972 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.018451 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.019221 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020182 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.020477 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020514 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.040162 (+ 19648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.040165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.040165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.040174 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.041997 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.042864 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.042865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.042865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.042959 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.043149 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.043149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.043149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.043303 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.043354 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.043503 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.043562 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:06:38.043684 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.017555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43464)
0924 23:06:38.017695 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.017708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.017813 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.018045 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.018046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.018280 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.018353 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.018355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.019303 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.019304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.020104 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.020105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.020584 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.020584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.020643 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.038747 (+ 18104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.038754 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.040032 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.042083 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.042084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.042084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.042771 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.043016 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.043017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.043017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.043083 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.043370 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.043370 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.043412 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.043487 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.043581 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 23:06:38.049685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6539
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 23:06:38.055835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37045
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38659
--webserver_port=42589
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:06:38.057479 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.045766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60894)
0924 23:06:38.045817 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.045828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.045864 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.046120 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.046121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.046237 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.046275 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.046277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.047164 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.047165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.047662 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.047664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.048144 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.048145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.048214 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.048582 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.048586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.054551 (+ 5965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.056313 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.056314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.056314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.056487 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.056703 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.056704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.056704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.056761 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.057093 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.057094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.057147 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.057211 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.057351 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:06:38.057943 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.045947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.0.98.60:39849)
0924 23:06:38.046001 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.046003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.046010 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:38.046040 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.046041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.046050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.046110 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.046349 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.047083 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.047769 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.048073 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.048141 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.054647 (+ 6506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.054649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.054649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.054657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.056234 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.056559 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.056560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.056560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.056632 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.056828 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.056829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.056829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.056989 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.057057 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.057720 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.057797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:06:38.058384 6477 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39849
}
attrs {
promote: true
}
}
dest_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a"
cas_config_opid_index: 5
I20260924 23:06:38.058655 6477 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } }
I20260924 23:06:38.059000 7035 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.059881 7035 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39849
I20260924 23:06:38.059970 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.060061 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.060559 7128 master.cc:537] Master@127.0.98.60:39849 shutting down...
I20260924 23:06:38.060652 7170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:38.061666 7170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Refusing update from remote peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 23:06:38.062119 7222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 23:06:38.063668 6463 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a -> Peer 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60:39849): Couldn't send request to peer 0b5b0ff144db47fcb0b5996a9537555d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 23:06:38.064673 7128 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:38.064765 7128 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:38.064819 7128 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: stopping tablet replica
I20260924 23:06:38.067195 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.060323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.0.98.61:40649)
0924 23:06:38.060368 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.060370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.063348 (+ 2978us) server_negotiation.cc:407] Connection header received
0924 23:06:38.063379 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.063380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.063412 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.063475 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.063835 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.064321 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.066047 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.066758 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.066944 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.067013 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.067098 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.067131 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:38.067203 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.060272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36404)
0924 23:06:38.061909 (+ 1637us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.061921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.061976 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.063538 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.063539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.063541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.063586 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.063588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.064602 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.064603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.065958 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.065962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.066952 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.066953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.067014 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.067034 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.067091 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.067131 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1588}
I20260924 23:06:38.067651 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 LEARNER]: Refusing update from remote peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 23:06:38.067840 7222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 23:06:38.068441 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.069634 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.069725 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: LEARNER
I20260924 23:06:38.069844 7036 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: attempting to promote NON_VOTER 6386922bf36145bf863f98aeb7dd9de3 to VOTER
I20260924 23:06:38.069671 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: true } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.069919 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: LEARNER
I20260924 23:06:38.070070 7036 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } }
I20260924 23:06:38.070568 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.070449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60898)
0924 23:06:38.070500 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.070538 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:06:38.070626 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 LEARNER]: Refusing update from remote peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 23:06:38.070791 7035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 23:06:38.071020 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.070938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60902)
0924 23:06:38.070968 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.070994 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 23:06:38.071192 6463 consensus_peers.cc:487] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a -> Peer 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60:39849): Couldn't send request to peer 0b5b0ff144db47fcb0b5996a9537555d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 23:06:38.071408 7222 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6386922bf36145bf863f98aeb7dd9de3 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.071545 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6386922bf36145bf863f98aeb7dd9de3 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.072299 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.072383 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.072499 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.072553 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.072669 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.072721 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:06:38.077719 7128 master.cc:552] Master@127.0.98.60:39849 shutdown complete.
I20260924 23:06:38.077791 7128 master_runner.cc:305] Clearing existing system tablet
I20260924 23:06:38.077976 7128 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:38.078037 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:38.078676 7128 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:38.079190 7227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.079339 7128 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:38.079396 7128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.079422 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
uuid: "0b5b0ff144db47fcb0b5996a9537555d"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "15378d1af74f9511cffd91954dc42f35"
server_key_iv: "9538768c2856d41e9a628eb55a2c2dad"
server_key_version: "encryptionkey@0"
I20260924 23:06:38.080168 7128 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:38.081471 7128 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 23:06:38.081550 7128 log.cc:1196] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 23:06:38.081743 7128 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Deleting consensus metadata
I20260924 23:06:38.082048 7128 master_runner.cc:315] Copying system tablet from 127.0.98.62:38659
I20260924 23:06:38.082651 7128 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38659
I20260924 23:06:38.089178 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.083007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.0.98.62:38659)
0924 23:06:38.083051 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.083053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.083194 (+ 141us) server_negotiation.cc:407] Connection header received
0924 23:06:38.083279 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.083279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.083291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.083355 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.083727 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.084217 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.085309 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.085512 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.085588 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.086600 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.086601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.086602 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.086609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.088023 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.088388 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.088389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.088389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.088447 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.088665 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.088666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.088666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.088774 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.088850 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.089011 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.089087 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:38.089187 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.082958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43478)
0924 23:06:38.083100 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.083110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.083196 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.083361 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.083362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.083591 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.083647 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.083649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.084302 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.084303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.085216 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.085218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.085587 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.085587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.085639 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.086195 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.086200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.086492 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.088125 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.088127 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.088127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.088311 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.088524 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.088524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.088525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.088587 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.088859 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.088859 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.088902 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.088985 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.089087 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260924 23:06:38.089705 6497 tablet_copy_service.cc:140] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0b5b0ff144db47fcb0b5996a9537555d ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43478)
I20260924 23:06:38.089780 6497 tablet_copy_service.cc:161] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0b5b0ff144db47fcb0b5996a9537555d at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43478: session id = 0b5b0ff144db47fcb0b5996a9537555d-00000000000000000000000000000000
I20260924 23:06:38.090338 6497 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 23:06:38.090808 7128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 23:06:38.091872 7128 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: tablet copy: Starting download of 0 data blocks...
I20260924 23:06:38.092021 7128 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: tablet copy: Starting download of 1 WAL segments...
I20260924 23:06:38.092676 7128 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 23:06:38.093385 6497 tablet_copy_service.cc:342] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Request end of tablet copy session 0b5b0ff144db47fcb0b5996a9537555d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43478
I20260924 23:06:38.093438 6497 tablet_copy_service.cc:433] P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: ending tablet copy session 0b5b0ff144db47fcb0b5996a9537555d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0b5b0ff144db47fcb0b5996a9537555d
I20260924 23:06:38.095927 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:38.097142 7234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:38.097347 7235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.099057 7128 server_base.cc:1034] running on GCE node
W20260924 23:06:38.106714 7237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.106941 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:38.107131 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:38.108254 7128 hybrid_clock.cc:648] HybridClock initialized: now 1790291198108239 us; error 38 us; skew 500 ppm
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:38.109779 7128 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 23:06:38.110150 7128 webserver.cc:460] Webserver started at http://127.0.98.60:41117/ using document root <none> and password file <none>
I20260924 23:06:38.110291 7128 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:38.110332 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:38.110958 7128 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:38.111476 7244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.111618 7128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:38.111665 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
uuid: "0b5b0ff144db47fcb0b5996a9537555d"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "15378d1af74f9511cffd91954dc42f35"
server_key_iv: "9538768c2856d41e9a628eb55a2c2dad"
server_key_version: "encryptionkey@0"
I20260924 23:06:38.111726 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:38.121706 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:38.123728 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:38.126703 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39849
I20260924 23:06:38.127048 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
I20260924 23:06:38.127337 7296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39849 every 8 connection(s)
I20260924 23:06:38.127899 7297 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:06:38.128304 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Bootstrap starting.
I20260924 23:06:38.130656 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:06:38.130893 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: Bootstrap complete.
I20260924 23:06:38.131147 7297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } }
I20260924 23:06:38.131246 7297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0b5b0ff144db47fcb0b5996a9537555d, State: Initialized, Role: LEARNER
I20260924 23:06:38.131294 7297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } }
I20260924 23:06:38.131474 7297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:38.131484 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.131708 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: This master's current role is: LEARNER
I20260924 23:06:38.132958 7311 catalog_manager.cc:1259] Loaded cluster ID: 90ab479602ae482a96c9c1a70ffee77a
I20260924 23:06:38.133040 7311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: loading cluster ID for follower catalog manager: success
I20260924 23:06:38.134158 7311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: acquiring CA information for follower catalog manager: success
I20260924 23:06:38.134438 7311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 23:06:38.173787 7221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:38.174088 7221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:38.174124 7221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:38.175731 7221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:38.175796 7221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:38.175835 7221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:38.177924 7221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42589
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:38.178162 7221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:38.178406 7221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:38.183852 7316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:38.183859 7317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:38.183868 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.184216 7221 server_base.cc:1034] running on GCE node
I20260924 23:06:38.184511 7221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:38.184820 7221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:38.185952 7221 hybrid_clock.cc:648] HybridClock initialized: now 1790291198185928 us; error 40 us; skew 500 ppm
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:38.189016 7221 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 23:06:38.189455 7221 webserver.cc:460] Webserver started at http://127.0.98.1:42589/ using document root <none> and password file <none>
I20260924 23:06:38.189705 7221 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:38.189796 7221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:38.191380 7221 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:38.193995 7326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.194169 7221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:38.194273 7221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "4b487ca706f14b5dabeb532009e2b2ed"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "2e2f46d1c30bf23b713a3706062be49f"
server_key_iv: "4829be74e26123cef39e53efc38d67d3"
server_key_version: "encryptionkey@0"
I20260924 23:06:38.194630 7221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:38.205425 7221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:38.207721 7221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:38.207831 7221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:38.208490 7221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:38.208804 7221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:38.208838 7221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.208860 7221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:38.208873 7221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.215032 7221 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37045
I20260924 23:06:38.215145 7439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37045 every 8 connection(s)
I20260924 23:06:38.215447 7221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:38.221896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 7221
I20260924 23:06:38.222057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6673
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:38.228130 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.216192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39843 (local address 127.0.98.61:40649)
0924 23:06:38.216241 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.216243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.216620 (+ 377us) server_negotiation.cc:407] Connection header received
0924 23:06:38.217657 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.217658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.217671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.217704 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.218481 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.219237 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220325 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.220602 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220653 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.224442 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.224444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.224444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.224452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.226354 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.226878 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.226879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.226879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.226939 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.227243 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.227244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.227245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.227399 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.227466 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.227953 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.228034 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:38.228130 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.216494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33089 (local address 127.0.98.62:38659)
0924 23:06:38.216541 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.216544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.216640 (+ 96us) server_negotiation.cc:407] Connection header received
0924 23:06:38.217657 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.217659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.217671 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.217704 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.218482 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.219231 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220699 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.220963 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.221020 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.224982 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.224983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.224984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.224993 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.226930 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.227326 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.227339 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.227340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.227389 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.227599 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.227600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.227601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.227721 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.227762 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.227951 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.228034 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:38.228672 7445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.216421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:33089)
0924 23:06:38.216544 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.216553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.217525 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.217810 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.217815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.218264 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.218369 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.218376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.219340 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.219341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.220251 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.220252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.221062 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.221064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.221130 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.221621 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.221625 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.224847 (+ 3222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.227013 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.227014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.227015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.227248 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.227466 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.227467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.227467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.227526 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.227807 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.227808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.227840 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.228215 (+ 375us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.228437 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":43,"thread_start_us":65,"threads_started":1}
I20260924 23:06:38.229045 7446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.216200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.98.1:39843)
0924 23:06:38.216472 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.216516 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.217525 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.217811 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.217815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.218256 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.218369 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.218376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.219329 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.219330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.220245 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.220247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.220675 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.220729 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.221422 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.221428 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.224288 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.226453 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.226459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.226461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.226767 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.227013 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.227014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.227015 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.227158 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.227610 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.227617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.227782 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.228657 (+ 875us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.228976 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":24,"thread_start_us":101,"threads_started":1}
I20260924 23:06:38.229451 7442 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40649
I20260924 23:06:38.229553 7442 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.229751 7442 heartbeater.cc:511] Master 127.0.98.61:40649 requested a full tablet report, sending...
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 23:06:38.230432 7441 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260924 23:06:38.230526 7441 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.230628 7441 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260924 23:06:38.231004 6477 ts_manager.cc:194] Re-registered known tserver with Master: 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045)
I20260924 23:06:38.231150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37683
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38659
--webserver_port=37883
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:06:38.231467 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33089
I20260924 23:06:38.234362 7074 ts_manager.cc:194] Registered new tserver with Master: 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045)
I20260924 23:06:38.235276 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.216211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50013 (local address 127.0.98.60:39849)
0924 23:06:38.216320 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.216321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.216655 (+ 334us) server_negotiation.cc:407] Connection header received
0924 23:06:38.217642 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.217643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.217657 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.217731 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.218465 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.219116 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220162 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.220447 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220517 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.230606 (+ 10089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.230608 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.230608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.230616 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.234011 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.234410 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.234411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.234412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.234474 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.234657 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.234658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.234658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.234781 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.234856 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.235078 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.235177 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260924 23:06:38.235738 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.216200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.98.1:50013)
0924 23:06:38.216534 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.216546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.217540 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.217817 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.217818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.218182 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.218369 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.218373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.219218 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.219219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.220058 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.220060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.220521 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.220521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.220615 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.221382 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.221394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.230504 (+ 9110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.234088 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.234089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.234090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.234318 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.234537 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.234537 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.234538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.234598 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.234857 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.234858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.234922 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.235049 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.235191 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260924 23:06:38.236418 7440 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39849
I20260924 23:06:38.236478 7440 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.236538 7440 heartbeater.cc:511] Master 127.0.98.60:39849 requested a full tablet report, sending...
I20260924 23:06:38.236974 7261 ts_manager.cc:194] Registered new tserver with Master: 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045)
W20260924 23:06:38.323103 7448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:38.323360 7448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:38.323385 7448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:38.324486 7448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:38.324520 7448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:38.324546 7448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:38.325989 7448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37883
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:38.326205 7448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:38.326376 7448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:38.331982 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:38.331964 7456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.332145 7448 server_base.cc:1034] running on GCE node
W20260924 23:06:38.331964 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.332396 7448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:38.332598 7448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:38.333743 7448 hybrid_clock.cc:648] HybridClock initialized: now 1790291198333723 us; error 35 us; skew 500 ppm
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:38.339881 7448 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 23:06:38.340355 7448 webserver.cc:460] Webserver started at http://127.0.98.2:37883/ using document root <none> and password file <none>
I20260924 23:06:38.340582 7448 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:38.340637 7448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:38.342127 7448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 23:06:38.342849 7463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.343129 7448 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:38.343240 7448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "61f6d6984b964a15a5498ce99dfd41f5"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "106fd69b8c579d65263328649abd4912"
server_key_iv: "4a169cc3a146a2a81937db46e9d2e0c3"
server_key_version: "encryptionkey@0"
I20260924 23:06:38.343601 7448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:38.354835 7448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:38.356736 7448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:38.356901 7448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:38.357132 7448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:38.357426 7448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:38.357481 7448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.357519 7448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:38.357548 7448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.363199 7448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37683
I20260924 23:06:38.363271 7576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37683 every 8 connection(s)
I20260924 23:06:38.363606 7448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:38.367620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 7448
I20260924 23:06:38.367770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6807
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 23:06:38.374042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46601
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38659
--webserver_port=41245
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:06:38.375495 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.364370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55985 (local address 127.0.98.62:38659)
0924 23:06:38.364433 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.364435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.364650 (+ 215us) server_negotiation.cc:407] Connection header received
0924 23:06:38.365387 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.365389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.365402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.365437 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.366206 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.366974 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.368044 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.368277 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.368315 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.372126 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.372127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.372128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.372135 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.373799 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.374324 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.374326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.374326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.374384 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.374680 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.374681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.374681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.374842 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.374895 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.375260 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.375349 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:38.376199 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.364334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:55985)
0924 23:06:38.364502 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.364542 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.365262 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.365527 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.365530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.365848 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.366088 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.366096 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.367063 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.367064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.367953 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.367955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.368398 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.368399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.368524 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.369066 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.369316 (+ 250us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.372033 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.373914 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.373921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.373924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.374227 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.374458 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.374459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.374459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.374600 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.374929 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.374935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.375101 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.375600 (+ 499us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.375940 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":51,"thread_start_us":95,"threads_started":1}
I20260924 23:06:38.376537 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.364720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54609 (local address 127.0.98.60:39849)
0924 23:06:38.364756 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.364757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.364862 (+ 105us) server_negotiation.cc:407] Connection header received
0924 23:06:38.365387 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.365389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.365406 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.365437 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.366204 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.366964 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.369135 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.369465 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.369530 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.372076 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.372077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.372078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.372085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.373847 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.375653 (+ 1806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.375655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.375655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.375718 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.375931 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.375932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.375933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.376069 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.376111 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.376323 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.376415 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:06:38.376611 7578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260924 23:06:38.376703 7578 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.376850 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.364645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.98.2:54609)
0924 23:06:38.364770 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.364779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.365264 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.365916 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.365917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.366061 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.366111 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.366112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.367063 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.367064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.369064 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.369067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.369582 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.369584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.369656 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.370494 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.370499 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.371975 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.375182 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.375182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.375183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.375554 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.375786 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.375786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.375787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.375854 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.376155 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.376155 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.376198 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.376641 (+ 443us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.376753 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":43,"thread_start_us":58,"threads_started":1}
I20260924 23:06:38.376916 7578 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260924 23:06:38.377132 7577 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39849
I20260924 23:06:38.377167 7577 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.377221 7577 heartbeater.cc:511] Master 127.0.98.60:39849 requested a full tablet report, sending...
I20260924 23:06:38.377260 6477 ts_manager.cc:194] Re-registered known tserver with Master: 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683)
I20260924 23:06:38.377468 7261 ts_manager.cc:194] Registered new tserver with Master: 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683)
I20260924 23:06:38.377764 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55985
I20260924 23:06:38.378304 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.365063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51939 (local address 127.0.98.61:40649)
0924 23:06:38.365142 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.365143 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.365206 (+ 63us) server_negotiation.cc:407] Connection header received
0924 23:06:38.365373 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.365374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.365387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.365447 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.366182 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.366673 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.368430 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.368756 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.368808 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.372875 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.372876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.372876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.372884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.375082 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.375945 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.375947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.375947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.375996 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.376235 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.376236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.376237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.376373 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.376415 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.378164 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.378240 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 23:06:38.378867 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.365006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.98.2:51939)
0924 23:06:38.365142 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.365157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.365267 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.365526 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.365529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.365827 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.366088 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.366092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.366779 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.366780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.367738 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.367741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.368881 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.368882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.368956 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.369344 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.369348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.372396 (+ 3048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.375169 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.375170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.375171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.375865 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.376106 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.376107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.376107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.376163 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.377982 (+ 1819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.377984 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.378048 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.378435 (+ 387us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.378554 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260924 23:06:38.379359 7579 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40649
I20260924 23:06:38.379427 7579 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.379498 7579 heartbeater.cc:511] Master 127.0.98.61:40649 requested a full tablet report, sending...
I20260924 23:06:38.379805 7074 ts_manager.cc:194] Registered new tserver with Master: 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683)
W20260924 23:06:38.457701 7584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:38.457948 7584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:38.457974 7584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:38.459062 7584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:38.459096 7584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:38.459122 7584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:38.460602 7584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41245
--tserver_master_addrs=127.0.98.62:38659,127.0.98.61:40649,127.0.98.60:39849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:38.460798 7584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:38.460987 7584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:38.466331 7592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.466437 7584 server_base.cc:1034] running on GCE node
W20260924 23:06:38.466533 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:38.466656 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:38.466921 7584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:38.467178 7584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:38.468371 7584 hybrid_clock.cc:648] HybridClock initialized: now 1790291198468367 us; error 38 us; skew 500 ppm
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:38.471055 7584 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 23:06:38.471483 7584 webserver.cc:460] Webserver started at http://127.0.98.3:41245/ using document root <none> and password file <none>
I20260924 23:06:38.471714 7584 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:38.471797 7584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:38.473148 7584 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:38.473747 7599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.473973 7584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:38.474064 7584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "4ae6392fb8a648d59312a7b169c22d0b"
format_stamp: "Formatted at 2026-09-24 23:06:37 on dist-test-slave-ph9j"
server_key: "0906ad9fc4a9f324658fef7b8d588e7d"
server_key_iv: "f394a16584508c90b598b5be26d805bf"
server_key_version: "encryptionkey@0"
I20260924 23:06:38.474396 7584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:38.485577 7584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:38.487877 7584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:38.488003 7584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:38.488190 7584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:38.488533 7584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:38.488570 7584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.488593 7584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:38.488628 7584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:38.494818 7584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46601
I20260924 23:06:38.494922 7712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46601 every 8 connection(s)
I20260924 23:06:38.495265 7584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:38.497974 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.494545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.0.98.60:39849)
0924 23:06:38.494596 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.494598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.494610 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:38.494694 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.494695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.494722 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.494813 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.494983 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.495658 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.497046 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.497626 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.497716 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.497793 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.497854 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.497916 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:06:38.497970 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.494485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60918)
0924 23:06:38.494558 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.494571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.494624 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.494831 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.494833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.494835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.494901 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.494904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.495765 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.495767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.496957 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.496961 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.497718 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.497719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.497775 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.497779 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.497815 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.497862 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 23:06:38.498649 7271 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 23:06:38.498871 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.498976 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: This master's current role is: LEARNER
I20260924 23:06:38.499955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 7584
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 23:06:38 dist-test-slave-ph9j krb5kdc[6432](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291198, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 23:06:38.510035 7716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.496129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:49687)
0924 23:06:38.496379 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.496389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.497032 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.497711 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.497712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.497944 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.498130 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.498149 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.499333 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.499334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.500692 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.500695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.501248 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.501249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.501406 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.502371 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.502382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.504236 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.508202 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.508210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.508212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.508516 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.508744 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.508745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.508746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.508909 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.509218 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.509225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.509404 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.509527 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.509712 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":87,"thread_start_us":87,"threads_started":1}
I20260924 23:06:38.510646 6528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.496300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49687 (local address 127.0.98.62:38659)
0924 23:06:38.496340 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.496343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.496435 (+ 92us) server_negotiation.cc:407] Connection header received
0924 23:06:38.497182 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.497183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.497195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.497243 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.498437 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.499180 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.500828 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.501119 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.501183 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.504336 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.504338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.504338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.504345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.508136 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.508602 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.508604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.508604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.508663 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.508994 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.508996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.508996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.509130 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.509191 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.510448 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.510533 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:38.510998 7714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260924 23:06:38.511101 7714 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.511304 7714 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260924 23:06:38.511654 6477 ts_manager.cc:194] Re-registered known tserver with Master: 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601)
I20260924 23:06:38.511826 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.501409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.0.98.60:39849)
0924 23:06:38.501443 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.501445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.501967 (+ 522us) server_negotiation.cc:407] Connection header received
0924 23:06:38.502023 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.502024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.502036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.502088 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.502952 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.503660 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.504851 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.505146 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.505185 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.507586 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.507587 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.507587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.507594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.509294 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.509692 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.509693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.509693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.509748 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.510017 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.510019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.510019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.510143 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.510905 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.511632 (+ 727us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.511708 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:06:38.512168 6477 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49687
I20260924 23:06:38.512531 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.496823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38961 (local address 127.0.98.61:40649)
0924 23:06:38.496876 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.496878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.496902 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:06:38.498224 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.498225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.498236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.498284 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.498730 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.499408 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.500821 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.501120 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.501160 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.509300 (+ 8140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.509301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.509301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.509307 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.511147 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.511744 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.511745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.511745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.511799 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.512011 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.512011 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.512012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.512123 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.512160 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.512359 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.512447 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:38.513379 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.496651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.98.3:38961)
0924 23:06:38.496782 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.496863 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.498099 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.498325 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.498326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.498565 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.498636 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.498637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.499506 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.499506 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.500701 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.500704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.501250 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.501250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.501406 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.505971 (+ 4565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.505977 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.508863 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.511348 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.511349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.511349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.511625 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.511886 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.511887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.511887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.511944 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.512205 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.512205 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.512247 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.512645 (+ 398us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.513262 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":544,"thread_start_us":76,"threads_started":1}
I20260924 23:06:38.513687 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.496236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38277 (local address 127.0.98.60:39849)
0924 23:06:38.496326 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.496327 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.496397 (+ 70us) server_negotiation.cc:407] Connection header received
0924 23:06:38.497183 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.497184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.497195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.497246 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.498240 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.499390 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.500854 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.501154 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.501209 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.509135 (+ 7926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.509136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.509137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.509144 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.511678 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.512361 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.512362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.512362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.512423 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.512836 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.512837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.512837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.512971 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.513036 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.513283 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.513368 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":38,"threads_started":1}
I20260924 23:06:38.513686 7715 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40649
I20260924 23:06:38.513731 7715 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.513797 7715 heartbeater.cc:511] Master 127.0.98.61:40649 requested a full tablet report, sending...
I20260924 23:06:38.514039 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.501031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.98.61:40649)
0924 23:06:38.501834 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.501837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.501945 (+ 108us) server_negotiation.cc:407] Connection header received
0924 23:06:38.502134 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.502134 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.502145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.502203 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.502968 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.503739 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.504874 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.505196 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.505258 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.507557 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.507558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.507559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.507566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.509976 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.512992 (+ 3016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.512993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.512994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.513048 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.513267 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.513267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.513268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.513384 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.513478 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.513811 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.513927 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":50,"threads_started":1}
I20260924 23:06:38.514154 7074 ts_manager.cc:194] Registered new tserver with Master: 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601)
I20260924 23:06:38.514592 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.495977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.98.3:38277)
0924 23:06:38.496273 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.496318 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.497032 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.497356 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.497361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.497810 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:38.498130 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.498136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.499456 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.499457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.500701 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.500704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.501257 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.501258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.501406 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.506792 (+ 5386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:38.506797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:38.509051 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:38.512010 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.512012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.512012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.512236 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.512500 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:38.512501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:38.512502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:38.512560 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:38.513064 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:38.513065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:38.513107 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.513177 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.514501 (+ 1324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1208,"thread_start_us":91,"threads_started":1}
I20260924 23:06:38.514885 7713 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39849
I20260924 23:06:38.514952 7713 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:38.515000 7713 heartbeater.cc:511] Master 127.0.98.60:39849 requested a full tablet report, sending...
I20260924 23:06:38.515249 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.501770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.0.98.62:38659)
0924 23:06:38.502213 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.502215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.502221 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:06:38.502240 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.502240 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.502253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.503068 (+ 815us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.503104 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.503806 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.504848 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.505147 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.505455 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.506936 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.506938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.506938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.506946 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.513691 (+ 6745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.514150 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.514151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.514151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.514215 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.514495 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.514496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.514496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.514653 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.515065 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.515107 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.515192 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":95,"threads_started":1}
I20260924 23:06:38.515254 7261 ts_manager.cc:194] Registered new tserver with Master: 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601)
I20260924 23:06:38.516606 6477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:38.518181 6477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:38.518939 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: true } } }
I20260924 23:06:38.519035 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.520113 7035 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: attempting to promote NON_VOTER 0b5b0ff144db47fcb0b5996a9537555d to VOTER
I20260924 23:06:38.520227 7035 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:38.521041 7270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 LEARNER]: Refusing update from remote peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 23:06:38.521265 7222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 23:06:38.521991 7035 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.522132 7270 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.522933 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.523012 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.523077 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.523126 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.523218 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.523291 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:06:38.527772 7084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Refusing update from remote peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 23:06:38.527896 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.523681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37045 (local address 127.0.0.1:33998)
0924 23:06:38.523810 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.523821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.523863 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.524359 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.524361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.524362 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.524416 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.524418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.525680 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.525681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.526750 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.526752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.527686 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.527688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.527751 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.527755 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.527785 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.527823 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 23:06:38.527997 7222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 23:06:38.528307 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.523950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.0.98.1:37045)
0924 23:06:38.524168 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.524172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.524182 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:38.524225 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.524228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.524246 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.524342 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.524511 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.525564 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.526851 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.527574 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.528073 (+ 499us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.528168 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.528197 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.528245 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
I20260924 23:06:38.528667 7084 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.528927 7374 tablet_service.cc:1467] Processing CreateTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d (DEFAULT_TABLE table=test-table [id=3de9821c87474c7f815e9541f86bea4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:38.529260 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010331c1e3194ebfbb0796f28b86ed0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:38.529654 7113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.529740 7113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:06:38.530194 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.523727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46601 (local address 127.0.0.1:52648)
0924 23:06:38.524259 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.524271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.524316 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.526093 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.526093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.526095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.526128 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.526129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.527313 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.527313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.528336 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.528339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.529983 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.529984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.530045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.530049 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.530079 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.530116 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":76,"threads_started":1}
I20260924 23:06:38.530195 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.523924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.98.2:37683)
0924 23:06:38.524080 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.524085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.524457 (+ 372us) server_negotiation.cc:407] Connection header received
0924 23:06:38.524520 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.524523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.524536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.524635 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.524729 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.525617 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.527262 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.529263 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.529320 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.529409 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.530074 (+ 665us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.530116 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260924 23:06:38.530457 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.523747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37683 (local address 127.0.0.1:40636)
0924 23:06:38.524401 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.524411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.524453 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.524633 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.524634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.524635 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.524676 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.524678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.525702 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.525702 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.527166 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.527169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.529858 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.529859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.529978 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.529983 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.530304 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.530382 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":48,"thread_start_us":62,"threads_started":1}
I20260924 23:06:38.530597 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.525148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.98.3:46601)
0924 23:06:38.525902 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.525905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.525915 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:38.525948 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.525950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.525964 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.526036 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.526199 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.527232 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.528430 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.529833 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.530356 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.530436 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.530460 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.530502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":142,"threads_started":1}
I20260924 23:06:38.531177 7511 tablet_service.cc:1467] Processing CreateTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d (DEFAULT_TABLE table=test-table [id=3de9821c87474c7f815e9541f86bea4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:38.531177 7647 tablet_service.cc:1467] Processing CreateTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d (DEFAULT_TABLE table=test-table [id=3de9821c87474c7f815e9541f86bea4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:38.531504 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010331c1e3194ebfbb0796f28b86ed0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:38.531522 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010331c1e3194ebfbb0796f28b86ed0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:38.532689 7742 tablet_bootstrap.cc:492] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Bootstrap starting.
I20260924 23:06:38.533425 7742 tablet_bootstrap.cc:654] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:38.534201 7742 log.cc:824] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:38.534861 7745 tablet_bootstrap.cc:492] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Bootstrap starting.
I20260924 23:06:38.535298 7742 tablet_bootstrap.cc:492] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: No bootstrap required, opened a new log
I20260924 23:06:38.535369 7742 ts_tablet_manager.cc:1397] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 23:06:38.535537 7745 tablet_bootstrap.cc:654] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:38.535830 7745 log.cc:824] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:38.536134 7744 tablet_bootstrap.cc:492] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Bootstrap starting.
I20260924 23:06:38.536490 7745 tablet_bootstrap.cc:492] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: No bootstrap required, opened a new log
I20260924 23:06:38.536574 7745 ts_tablet_manager.cc:1397] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:38.536707 7744 tablet_bootstrap.cc:654] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:38.536993 7744 log.cc:824] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:38.537803 7744 tablet_bootstrap.cc:492] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: No bootstrap required, opened a new log
I20260924 23:06:38.537873 7744 ts_tablet_manager.cc:1397] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:38.537981 7742 raft_consensus.cc:348] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.538141 7742 raft_consensus.cc:374] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:38.538175 7742 raft_consensus.cc:729] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b487ca706f14b5dabeb532009e2b2ed, State: Initialized, Role: FOLLOWER
I20260924 23:06:38.538313 7742 consensus_queue.cc:260] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.538558 7742 ts_tablet_manager.cc:1428] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 23:06:38.538686 7441 heartbeater.cc:503] Master 127.0.98.62:38659 was elected leader, sending a full tablet report...
I20260924 23:06:38.539006 7745 raft_consensus.cc:348] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.539142 7745 raft_consensus.cc:374] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:38.539172 7745 raft_consensus.cc:729] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f6d6984b964a15a5498ce99dfd41f5, State: Initialized, Role: FOLLOWER
I20260924 23:06:38.539288 7745 consensus_queue.cc:260] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.539546 7745 ts_tablet_manager.cc:1428] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:06:38.539821 7578 heartbeater.cc:503] Master 127.0.98.62:38659 was elected leader, sending a full tablet report...
I20260924 23:06:38.539928 7744 raft_consensus.cc:348] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.540078 7744 raft_consensus.cc:374] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:38.540119 7744 raft_consensus.cc:729] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae6392fb8a648d59312a7b169c22d0b, State: Initialized, Role: FOLLOWER
I20260924 23:06:38.540223 7744 consensus_queue.cc:260] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.540508 7744 ts_tablet_manager.cc:1428] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:06:38.540665 7714 heartbeater.cc:503] Master 127.0.98.62:38659 was elected leader, sending a full tablet report...
W20260924 23:06:38.615216 7580 tablet.cc:2369] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:06:38.716357 7443 tablet.cc:2369] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:06:38.746876 7717 tablet.cc:2369] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:38.765944 7750 raft_consensus.cc:482] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:38.766072 7750 raft_consensus.cc:504] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.766554 7750 leader_election.cc:288] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045), 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683)
I20260924 23:06:38.769413 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.766792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60467 (local address 127.0.98.1:37045)
0924 23:06:38.766842 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.766844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.766854 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:38.766974 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.766974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.766993 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.767059 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.767216 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.767688 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.768665 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.769073 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.769156 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.769166 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.769310 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.769351 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:06:38.769413 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.766714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37045 (local address 127.0.98.3:60467)
0924 23:06:38.766768 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.766781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.766880 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.767060 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.767061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.767063 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.767135 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.767137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.767784 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.767785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.768569 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.768571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.769154 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.769155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.769225 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.769229 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.769303 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.769344 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":3}
I20260924 23:06:38.769567 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.766813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42155 (local address 127.0.98.2:37683)
0924 23:06:38.766857 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.766859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.766869 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:38.766943 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.766944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.766957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.767036 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:38.767202 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.767674 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.768665 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.769144 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.769216 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.769225 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.769448 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.769485 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:38.769616 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.766756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37683 (local address 127.0.98.3:42155)
0924 23:06:38.766824 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:38.766835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:38.766879 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:38.767047 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:38.767048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:38.767049 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:38.767125 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.767127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.767760 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.767761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.768568 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:38.768570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.769257 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.769258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:38.769334 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.769338 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:38.769383 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:06:38.769418 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":66}
I20260924 23:06:38.769718 7394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010331c1e3194ebfbb0796f28b86ed0d" candidate_uuid: "4ae6392fb8a648d59312a7b169c22d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b487ca706f14b5dabeb532009e2b2ed" is_pre_election: true
I20260924 23:06:38.769923 7394 raft_consensus.cc:2454] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae6392fb8a648d59312a7b169c22d0b in term 0.
I20260924 23:06:38.769969 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010331c1e3194ebfbb0796f28b86ed0d" candidate_uuid: "4ae6392fb8a648d59312a7b169c22d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6d6984b964a15a5498ce99dfd41f5" is_pre_election: true
I20260924 23:06:38.770102 7531 raft_consensus.cc:2454] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae6392fb8a648d59312a7b169c22d0b in term 0.
I20260924 23:06:38.770143 7601 leader_election.cc:302] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae6392fb8a648d59312a7b169c22d0b, 4b487ca706f14b5dabeb532009e2b2ed; no voters:
I20260924 23:06:38.770262 7750 raft_consensus.cc:2781] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:38.770311 7750 raft_consensus.cc:482] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:38.770347 7750 raft_consensus.cc:3037] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:38.771046 7750 raft_consensus.cc:504] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.771189 7750 leader_election.cc:288] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [CANDIDATE]: Term 1 election: Requested vote from peers 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045), 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683)
I20260924 23:06:38.771320 7394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010331c1e3194ebfbb0796f28b86ed0d" candidate_uuid: "4ae6392fb8a648d59312a7b169c22d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b487ca706f14b5dabeb532009e2b2ed"
I20260924 23:06:38.771340 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010331c1e3194ebfbb0796f28b86ed0d" candidate_uuid: "4ae6392fb8a648d59312a7b169c22d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6d6984b964a15a5498ce99dfd41f5"
I20260924 23:06:38.771422 7531 raft_consensus.cc:3037] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:38.771430 7394 raft_consensus.cc:3037] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:38.772123 7394 raft_consensus.cc:2454] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae6392fb8a648d59312a7b169c22d0b in term 1.
I20260924 23:06:38.772411 7601 leader_election.cc:302] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae6392fb8a648d59312a7b169c22d0b, 4b487ca706f14b5dabeb532009e2b2ed; no voters:
I20260924 23:06:38.772508 7750 raft_consensus.cc:2781] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:38.772678 7750 raft_consensus.cc:686] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 1 LEADER]: Becoming Leader. State: Replica: 4ae6392fb8a648d59312a7b169c22d0b, State: Running, Role: LEADER
I20260924 23:06:38.772729 7531 raft_consensus.cc:2454] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae6392fb8a648d59312a7b169c22d0b in term 1.
I20260924 23:06:38.772797 7750 consensus_queue.cc:237] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } }
I20260924 23:06:38.773692 6477 catalog_manager.cc:5495] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4ae6392fb8a648d59312a7b169c22d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:38.774453 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:38.774554 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:38.810328 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.807733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.0.98.3:46601)
0924 23:06:38.807805 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.807807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.807824 (+ 17us) server_negotiation.cc:407] Connection header received
0924 23:06:38.807872 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.807873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.807893 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.807956 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:38.808175 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.808638 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.809450 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.809647 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.809710 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.809959 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:38.810064 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:38.810128 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.810167 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.810184 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 23:06:38.812294 7531 raft_consensus.cc:1261] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Refusing update from remote peer 4ae6392fb8a648d59312a7b169c22d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:38.812309 7394 raft_consensus.cc:1261] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Refusing update from remote peer 4ae6392fb8a648d59312a7b169c22d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:38.812712 7751 consensus_queue.cc:1035] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:38.812891 7750 consensus_queue.cc:1035] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:38.815272 7754 mvcc.cc:205] Tried to move back new op lower bound from 7333032750332239872 to 7333032750173908992. Current Snapshot: MvccSnapshot[applied={T|T < 7333032750332239872}]
I20260924 23:06:38.815513 7755 mvcc.cc:205] Tried to move back new op lower bound from 7333032750332239872 to 7333032750173908992. Current Snapshot: MvccSnapshot[applied={T|T < 7333032750332239872}]
I20260924 23:06:38.817724 6477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:38.817837 6477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:38.819538 6477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3c2dd1a93dd4b0eb6e0ef7991a4ec6a: Sending DeleteTablet for 3 replicas of tablet 010331c1e3194ebfbb0796f28b86ed0d
I20260924 23:06:38.819810 7374 tablet_service.cc:1514] Processing DeleteTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33998
I20260924 23:06:38.819851 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40636
I20260924 23:06:38.820055 7762 tablet_replica.cc:330] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: stopping tablet replica
I20260924 23:06:38.820391 7646 tablet_service.cc:1514] Processing DeleteTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52648
I20260924 23:06:38.820561 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet 010331c1e3194ebfbb0796f28b86ed0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:38 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52648
I20260924 23:06:38.820670 7765 tablet_replica.cc:330] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: stopping tablet replica
I20260924 23:06:38.820760 7765 raft_consensus.cc:2229] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:38.820878 7765 raft_consensus.cc:2258] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:38.821035 7762 raft_consensus.cc:2229] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:38.821388 7762 raft_consensus.cc:2258] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:38.821908 7765 ts_tablet_manager.cc:1905] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:38.822203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6445
I20260924 23:06:38.823942 7765 ts_tablet_manager.cc:1918] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:38.824024 7765 log.cc:1196] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/010331c1e3194ebfbb0796f28b86ed0d
I20260924 23:06:38.824291 7765 ts_tablet_manager.cc:1939] T 010331c1e3194ebfbb0796f28b86ed0d P 4ae6392fb8a648d59312a7b169c22d0b: Deleting consensus metadata
I20260924 23:06:38.825628 7762 ts_tablet_manager.cc:1905] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:38.827205 7762 ts_tablet_manager.cc:1918] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:38.827281 7762 log.cc:1196] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/010331c1e3194ebfbb0796f28b86ed0d
I20260924 23:06:38.827556 7762 ts_tablet_manager.cc:1939] T 010331c1e3194ebfbb0796f28b86ed0d P 4b487ca706f14b5dabeb532009e2b2ed: Deleting consensus metadata
I20260924 23:06:38.828737 7763 tablet_replica.cc:330] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: stopping tablet replica
I20260924 23:06:38.828922 7763 raft_consensus.cc:2229] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 23:06:38.829003 7601 connection.cc:535] server connection from 127.0.0.1:52648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:06:38.829155 7763 raft_consensus.cc:2258] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 23:06:38.829185 7328 connection.cc:535] server connection from 127.0.0.1:33998 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:06:38.829734 7763 ts_tablet_manager.cc:1905] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:38.831696 7763 ts_tablet_manager.cc:1918] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:38.831807 7719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.820685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.0.98.60:39849)
0924 23:06:38.820727 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.820729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.821954 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 23:06:38.823432 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.823433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.823446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.823478 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.824621 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.825231 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.826588 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.826889 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.826936 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.828653 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.828655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.828655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.828663 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.830436 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.830865 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.830867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.830867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.830926 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.831140 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.831141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.831142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.831270 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.831345 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.831617 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.831705 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:38.831816 7763 log.cc:1196] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/010331c1e3194ebfbb0796f28b86ed0d
I20260924 23:06:38.832960 7763 ts_tablet_manager.cc:1939] T 010331c1e3194ebfbb0796f28b86ed0d P 61f6d6984b964a15a5498ce99dfd41f5: Deleting consensus metadata
W20260924 23:06:38.833349 7467 connection.cc:441] server connection from 127.0.0.1:40636 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40636 (request call id 1) could send its response
I20260924 23:06:38.840327 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.831188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.0.98.61:40649)
0924 23:06:38.831243 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.831245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.832014 (+ 769us) server_negotiation.cc:407] Connection header received
0924 23:06:38.832134 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.832135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.832148 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.832179 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.834218 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.834799 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.835502 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.835735 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.835815 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.837322 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.837323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.837324 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.837333 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.839087 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.839454 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.839455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.839456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.839515 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.839716 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.839717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.839717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.839859 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.839957 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.840142 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.840221 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:06:38.840695 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:38.831544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.98.60:39849)
0924 23:06:38.831605 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:38.831607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:38.833134 (+ 1527us) server_negotiation.cc:407] Connection header received
0924 23:06:38.833485 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:38.833486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:38.833496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:38.833525 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:38.834268 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.835019 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.836144 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:38.836421 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:38.836514 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:38.837868 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:38.837870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:38.837870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:38.837903 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:38.839442 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.839788 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.839789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.839789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.839847 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:38.840048 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:38.840049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:38.840049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:38.840172 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:38.840258 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:38.840525 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 23:06:38.840600 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:39.816992 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:39.816676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:55527)
0924 23:06:39.816810 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:39.816928 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
W20260924 23:06:39.817205 7578 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:39.820336 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:39.820068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:44757)
0924 23:06:39.820212 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:39.820296 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260924 23:06:39.820538 7714 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:39.822243 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:39.822041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:42401)
0924 23:06:39.822137 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:39.822177 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":49,"threads_started":1}
W20260924 23:06:39.822484 7441 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.319727 7777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a)
I20260924 23:06:40.319933 7777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:40.320236 7777 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659), 6386922bf36145bf863f98aeb7dd9de3 (127.0.98.61:40649)
I20260924 23:06:40.320653 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.320377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43960)
0924 23:06:40.320512 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.320568 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260924 23:06:40.320798 7781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a)
W20260924 23:06:40.320822 7246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.320916 7781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:40.321267 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.321151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43974)
0924 23:06:40.321211 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.321237 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 23:06:40.321435 7246 leader_election.cc:334] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.321590 7781 leader_election.cc:288] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659), 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60:39849)
I20260924 23:06:40.321614 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.321274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:43984)
0924 23:06:40.321464 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.321572 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":60,"threads_started":1}
W20260924 23:06:40.321803 7059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.322367 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.322223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44004)
0924 23:06:40.322319 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.322341 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
W20260924 23:06:40.322429 7059 leader_election.cc:334] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.324148 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.320430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40649 (local address 127.0.0.1:36440)
0924 23:06:40.320635 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.320647 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:40.320718 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:40.320966 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:40.320967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:40.320969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:40.321025 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:40.321028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.322540 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.322541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:40.323297 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:40.323299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.323878 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.323879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:40.323955 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:40.323963 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 23:06:40.324036 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 23:06:40.324080 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260924 23:06:40.324146 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.320507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.98.61:40649)
0924 23:06:40.320638 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:40.320640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:40.320710 (+ 70us) server_negotiation.cc:407] Connection header received
0924 23:06:40.320799 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:40.320800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:40.320811 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:40.320905 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:40.321113 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.322455 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.323410 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.323787 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.323895 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:40.323908 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:40.324045 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 23:06:40.324083 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260924 23:06:40.324335 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.321497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.0.98.60:39849)
0924 23:06:40.321705 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:40.321707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:40.321716 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:40.321740 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:40.321741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:40.321751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:40.321822 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:40.322024 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.322688 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.323632 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.323990 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.324069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:40.324080 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:40.324229 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 23:06:40.324268 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260924 23:06:40.324332 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.321275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39849 (local address 127.0.0.1:60968)
0924 23:06:40.321558 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.321568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:40.321616 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:40.321868 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:40.321868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:40.321870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:40.321938 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:40.321940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.322793 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.322795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:40.323533 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:40.323535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:40.324068 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:40.324069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:40.324135 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:40.324139 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:40.324220 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 23:06:40.324261 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260924 23:06:40.324692 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b5b0ff144db47fcb0b5996a9537555d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6386922bf36145bf863f98aeb7dd9de3" is_pre_election: true
I20260924 23:06:40.324790 7084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b5b0ff144db47fcb0b5996a9537555d in term 1.
I20260924 23:06:40.324743 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6386922bf36145bf863f98aeb7dd9de3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b5b0ff144db47fcb0b5996a9537555d" is_pre_election: true
I20260924 23:06:40.324856 7270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6386922bf36145bf863f98aeb7dd9de3 in term 1.
I20260924 23:06:40.325035 7060 leader_election.cc:302] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b5b0ff144db47fcb0b5996a9537555d, 6386922bf36145bf863f98aeb7dd9de3; no voters: d3c2dd1a93dd4b0eb6e0ef7991a4ec6a
I20260924 23:06:40.325069 7248 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b5b0ff144db47fcb0b5996a9537555d, 6386922bf36145bf863f98aeb7dd9de3; no voters: d3c2dd1a93dd4b0eb6e0ef7991a4ec6a
I20260924 23:06:40.325163 7781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 23:06:40.325165 7777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 23:06:40.325207 7781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a)
I20260924 23:06:40.325207 7777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Starting leader election (detected failure of leader d3c2dd1a93dd4b0eb6e0ef7991a4ec6a)
I20260924 23:06:40.325233 7781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:06:40.325237 7777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:06:40.325929 7781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:40.325953 7777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:40.326074 7781 leader_election.cc:288] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 2 election: Requested vote from peers d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659), 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60:39849)
I20260924 23:06:40.326086 7777 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [CANDIDATE]: Term 2 election: Requested vote from peers d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659), 6386922bf36145bf863f98aeb7dd9de3 (127.0.98.61:40649)
I20260924 23:06:40.326229 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.326140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44018)
0924 23:06:40.326178 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.326204 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 23:06:40.326244 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6386922bf36145bf863f98aeb7dd9de3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b5b0ff144db47fcb0b5996a9537555d"
I20260924 23:06:40.326267 7084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b5b0ff144db47fcb0b5996a9537555d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6386922bf36145bf863f98aeb7dd9de3"
I20260924 23:06:40.326254 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.326168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44022)
0924 23:06:40.326206 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.326230 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 23:06:40.326329 7084 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0b5b0ff144db47fcb0b5996a9537555d in current term 2: Already voted for candidate 6386922bf36145bf863f98aeb7dd9de3 in this term.
I20260924 23:06:40.326331 7270 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6386922bf36145bf863f98aeb7dd9de3 in current term 2: Already voted for candidate 0b5b0ff144db47fcb0b5996a9537555d in this term.
I20260924 23:06:40.326690 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.326569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44032)
0924 23:06:40.326640 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.326670 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 23:06:40.326748 7246 leader_election.cc:334] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.326797 7246 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b5b0ff144db47fcb0b5996a9537555d; no voters: 6386922bf36145bf863f98aeb7dd9de3, d3c2dd1a93dd4b0eb6e0ef7991a4ec6a
I20260924 23:06:40.326886 7777 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 23:06:40.327127 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.327023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44048)
0924 23:06:40.327074 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.327105 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:06:40.327210 7059 leader_election.cc:334] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:40.327246 7059 leader_election.cc:302] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6386922bf36145bf863f98aeb7dd9de3; no voters: 0b5b0ff144db47fcb0b5996a9537555d, d3c2dd1a93dd4b0eb6e0ef7991a4ec6a
I20260924 23:06:40.327329 7781 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 23:06:40.817814 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.817546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:44481)
0924 23:06:40.817704 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.817753 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260924 23:06:40.821019 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.820827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:47365)
0924 23:06:40.820946 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.820983 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":55,"threads_started":1}
I20260924 23:06:40.823012 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:40.822749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:49219)
0924 23:06:40.822938 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:40.822973 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":103,"threads_started":1}
I20260924 23:06:41.818534 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:41.818271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:35631)
0924 23:06:41.818415 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:41.818475 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260924 23:06:41.821636 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:41.821426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:47415)
0924 23:06:41.821560 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:41.821600 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 23:06:41.823593 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:41.823423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:41473)
0924 23:06:41.823513 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:41.823549 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":41,"threads_started":1}
I20260924 23:06:42.257442 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:42.257544 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:42.257709 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659), 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60:39849)
I20260924 23:06:42.257998 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6386922bf36145bf863f98aeb7dd9de3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b5b0ff144db47fcb0b5996a9537555d" is_pre_election: true
I20260924 23:06:42.258076 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.257834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44298)
0924 23:06:42.258009 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.258042 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":102,"threads_started":1}
I20260924 23:06:42.258106 7270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6386922bf36145bf863f98aeb7dd9de3 in term 2.
I20260924 23:06:42.258291 7060 leader_election.cc:302] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5b0ff144db47fcb0b5996a9537555d, 6386922bf36145bf863f98aeb7dd9de3; no voters:
I20260924 23:06:42.258407 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 23:06:42.258456 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.258380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44312)
0924 23:06:42.258410 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.258435 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 23:06:42.258457 7794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 23:06:42.258558 7059 leader_election.cc:334] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:42.258548 7794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 2 FOLLOWER]: Advancing to term 3
I20260924 23:06:42.259339 7794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:42.259476 7794 leader_election.cc:288] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 3 election: Requested vote from peers d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659), 0b5b0ff144db47fcb0b5996a9537555d (127.0.98.60:39849)
I20260924 23:06:42.259641 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.259545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44318)
0924 23:06:42.259579 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.259612 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 23:06:42.259615 7270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6386922bf36145bf863f98aeb7dd9de3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b5b0ff144db47fcb0b5996a9537555d"
I20260924 23:06:42.259690 7270 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 2 FOLLOWER]: Advancing to term 3
I20260924 23:06:42.260337 7270 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6386922bf36145bf863f98aeb7dd9de3 in term 3.
I20260924 23:06:42.260600 7060 leader_election.cc:302] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5b0ff144db47fcb0b5996a9537555d, 6386922bf36145bf863f98aeb7dd9de3; no voters:
I20260924 23:06:42.260646 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.260555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44328)
0924 23:06:42.260605 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.260629 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:06:42.260712 7059 leader_election.cc:334] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
I20260924 23:06:42.260751 7794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 23:06:42.260869 7794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [term 3 LEADER]: Becoming Leader. State: Replica: 6386922bf36145bf863f98aeb7dd9de3, State: Running, Role: LEADER
I20260924 23:06:42.260959 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } }
I20260924 23:06:42.261322 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6386922bf36145bf863f98aeb7dd9de3. Latest consensus state: current_term: 3 leader_uuid: "6386922bf36145bf863f98aeb7dd9de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:42.261416 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:42.261595 7799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:42.262084 7799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:42.262145 7799 catalog_manager.cc:1259] Loaded cluster ID: 90ab479602ae482a96c9c1a70ffee77a
I20260924 23:06:42.262197 7799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:42.262383 7799 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:42.262477 7799 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Loaded TSK: 0
I20260924 23:06:42.262626 7799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:06:42.328897 7072 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:42.330705 7072 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:42.331915 7270 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [term 3 FOLLOWER]: Refusing update from remote peer 6386922bf36145bf863f98aeb7dd9de3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 23:06:42.332003 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.331785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44352)
0924 23:06:42.331906 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.331945 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 23:06:42.332163 7797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 23:06:42.332435 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.332344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.0.1:44360)
0924 23:06:42.332383 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.332410 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 23:06:42.332540 7059 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 -> Peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a (127.0.98.62:38659): Couldn't send request to peer d3c2dd1a93dd4b0eb6e0ef7991a4ec6a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 23:06:42.332864 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6386922bf36145bf863f98aeb7dd9de3. Latest consensus state: current_term: 3 leader_uuid: "6386922bf36145bf863f98aeb7dd9de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:42.332988 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:06:42.333480 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "6386922bf36145bf863f98aeb7dd9de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:42.333580 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:42.333745 7802 mvcc.cc:205] Tried to move back new op lower bound from 7333032764749029376 to 7333032764461539328. Current Snapshot: MvccSnapshot[applied={T|T < 7333032764749029376}]
I20260924 23:06:42.333899 7801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "6386922bf36145bf863f98aeb7dd9de3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d3c2dd1a93dd4b0eb6e0ef7991a4ec6a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } peers { permanent_uuid: "6386922bf36145bf863f98aeb7dd9de3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40649 } attrs { promote: false } } peers { permanent_uuid: "0b5b0ff144db47fcb0b5996a9537555d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39849 } attrs { promote: false } } }
I20260924 23:06:42.334023 7801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5b0ff144db47fcb0b5996a9537555d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:06:42.338903 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.335254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46601 (local address 127.0.0.1:52660)
0924 23:06:42.335426 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.335436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:42.335483 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:42.335919 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:42.335920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:42.335921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:42.335991 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.335992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.336928 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.336929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.337656 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.337657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.338587 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.338588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.338663 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.338669 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:06:42.338703 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 23:06:42.338760 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":21,"thread_start_us":86,"threads_started":1}
I20260924 23:06:42.338946 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.335556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.0.98.2:37683)
0924 23:06:42.335647 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:42.335649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:42.335655 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:06:42.335676 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:42.335677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:42.335685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:42.335757 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:42.335894 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.336486 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.337817 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.338505 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.338569 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.338582 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:42.338836 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 23:06:42.338879 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":50,"threads_started":1}
I20260924 23:06:42.339372 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.335208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37683 (local address 127.0.0.1:40642)
0924 23:06:42.335372 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.335381 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:42.335430 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:42.335759 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:42.335760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:42.335762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:42.335816 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.335818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.336571 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.336572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.337629 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.337631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.338612 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.338613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.338674 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.338678 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:42.338725 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:06:42.338769 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":28,"thread_start_us":72,"threads_started":1}
I20260924 23:06:42.339385 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.335651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.0.98.1:37045)
0924 23:06:42.335764 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:42.335766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:42.335776 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:42.335805 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:42.335806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:42.335817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:42.336057 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:42.336095 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.336823 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.338171 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.338983 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.339105 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.339143 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:42.339268 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 23:06:42.339310 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260924 23:06:42.339390 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.335208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37045 (local address 127.0.0.1:34014)
0924 23:06:42.335365 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.335376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:42.335430 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:42.335919 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:42.335920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:42.335921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:42.335991 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.335992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.336906 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.336907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.338053 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.338055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.339109 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.339110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.339172 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.339175 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:42.339241 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 23:06:42.339281 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":24}
I20260924 23:06:42.339668 7511 tablet_service.cc:1467] Processing CreateTablet for tablet e18ef837a3e04c9f9c5608e6ca0e44a0 (DEFAULT_TABLE table=test-table [id=97f50e08ce824982ba45d79d79c5cc79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:42.339666 7374 tablet_service.cc:1467] Processing CreateTablet for tablet e18ef837a3e04c9f9c5608e6ca0e44a0 (DEFAULT_TABLE table=test-table [id=97f50e08ce824982ba45d79d79c5cc79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:42.339794 7511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18ef837a3e04c9f9c5608e6ca0e44a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:42.339794 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18ef837a3e04c9f9c5608e6ca0e44a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:42.339811 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.335350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.98.3:46601)
0924 23:06:42.335779 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:42.335782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:42.335790 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:42.335823 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:42.335824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:42.335835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:42.335963 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:42.336083 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.336826 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.337746 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.338474 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.339665 (+ 1191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.339682 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:42.339699 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 23:06:42.339742 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":56,"threads_started":1}
I20260924 23:06:42.340022 7647 tablet_service.cc:1467] Processing CreateTablet for tablet e18ef837a3e04c9f9c5608e6ca0e44a0 (DEFAULT_TABLE table=test-table [id=97f50e08ce824982ba45d79d79c5cc79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:42.340140 7647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18ef837a3e04c9f9c5608e6ca0e44a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:42.341202 7811 tablet_bootstrap.cc:492] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Bootstrap starting.
I20260924 23:06:42.341653 7811 tablet_bootstrap.cc:654] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:42.341706 7812 tablet_bootstrap.cc:492] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Bootstrap starting.
I20260924 23:06:42.341981 7813 tablet_bootstrap.cc:492] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed: Bootstrap starting.
I20260924 23:06:42.342173 7812 tablet_bootstrap.cc:654] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:42.342370 7811 tablet_bootstrap.cc:492] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: No bootstrap required, opened a new log
I20260924 23:06:42.342379 7813 tablet_bootstrap.cc:654] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:42.342439 7811 ts_tablet_manager.cc:1397] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:42.342563 7811 raft_consensus.cc:348] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.342686 7811 raft_consensus.cc:374] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:42.342727 7811 raft_consensus.cc:729] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae6392fb8a648d59312a7b169c22d0b, State: Initialized, Role: FOLLOWER
I20260924 23:06:42.342803 7811 consensus_queue.cc:260] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.342845 7812 tablet_bootstrap.cc:492] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: No bootstrap required, opened a new log
I20260924 23:06:42.342898 7812 ts_tablet_manager.cc:1397] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:42.342983 7811 ts_tablet_manager.cc:1428] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:42.343039 7812 raft_consensus.cc:348] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.343175 7812 raft_consensus.cc:374] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:42.343243 7812 raft_consensus.cc:729] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f6d6984b964a15a5498ce99dfd41f5, State: Initialized, Role: FOLLOWER
I20260924 23:06:42.343348 7812 consensus_queue.cc:260] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.343387 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.343162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:54707)
0924 23:06:42.343317 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.343354 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260924 23:06:42.343591 7812 ts_tablet_manager.cc:1428] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:42.343672 7715 heartbeater.cc:503] Master 127.0.98.61:40649 was elected leader, sending a full tablet report...
I20260924 23:06:42.344069 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.343548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:46415)
0924 23:06:42.343654 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.343677 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260924 23:06:42.344398 7579 heartbeater.cc:503] Master 127.0.98.61:40649 was elected leader, sending a full tablet report...
I20260924 23:06:42.344549 7813 tablet_bootstrap.cc:492] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed: No bootstrap required, opened a new log
I20260924 23:06:42.344612 7813 ts_tablet_manager.cc:1397] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:06:42.344760 7813 raft_consensus.cc:348] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.344843 7813 raft_consensus.cc:374] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:42.344858 7813 raft_consensus.cc:729] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b487ca706f14b5dabeb532009e2b2ed, State: Initialized, Role: FOLLOWER
I20260924 23:06:42.344913 7813 consensus_queue.cc:260] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.345044 7813 ts_tablet_manager.cc:1428] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:42.345413 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.343951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:44823)
0924 23:06:42.345331 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.345368 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":89,"threads_started":1}
I20260924 23:06:42.345454 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.345260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:41771)
0924 23:06:42.345380 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.345406 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260924 23:06:42.345459 7442 heartbeater.cc:503] Master 127.0.98.61:40649 was elected leader, sending a full tablet report...
I20260924 23:06:42.345799 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.345696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:58775)
0924 23:06:42.345746 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.345769 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:06:42.429829 7822 raft_consensus.cc:482] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:42.429996 7822 raft_consensus.cc:504] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.430382 7822 leader_election.cc:288] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683), 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601)
I20260924 23:06:42.434397 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.430695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45865 (local address 127.0.98.3:46601)
0924 23:06:42.430737 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:42.430739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:42.430825 (+ 86us) server_negotiation.cc:407] Connection header received
0924 23:06:42.430893 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:42.430894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:42.430905 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:42.430957 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:42.431125 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.431893 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.433470 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.434048 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.434122 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.434133 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:42.434293 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 23:06:42.434329 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:42.434397 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.430619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34217 (local address 127.0.98.2:37683)
0924 23:06:42.430671 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:42.430673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:42.430684 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:42.430868 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:42.430870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:42.430883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:42.430930 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:42.431112 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.431895 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.433465 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.434048 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.434122 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.434134 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:42.434293 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 23:06:42.434328 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:06:42.434453 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.430578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46601 (local address 127.0.98.1:45865)
0924 23:06:42.430746 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.430756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:42.430816 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:42.430959 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:42.430959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:42.430961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:42.431069 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.431071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.432004 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.432005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.433375 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.433378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.434133 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.434135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.434209 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.434213 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:42.434273 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:06:42.434329 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":107,"thread_start_us":115,"threads_started":1}
I20260924 23:06:42.434442 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.430548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37683 (local address 127.0.98.1:34217)
0924 23:06:42.430613 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.430627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:42.430687 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:42.430951 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:42.430952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:42.430956 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:42.431048 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.431051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.431984 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.431985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.433375 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:42.433378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.434154 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.434155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:42.434216 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.434219 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:42.434274 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:06:42.434328 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":45}
I20260924 23:06:42.434823 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18ef837a3e04c9f9c5608e6ca0e44a0" candidate_uuid: "4b487ca706f14b5dabeb532009e2b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae6392fb8a648d59312a7b169c22d0b" is_pre_election: true
I20260924 23:06:42.434923 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18ef837a3e04c9f9c5608e6ca0e44a0" candidate_uuid: "4b487ca706f14b5dabeb532009e2b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6d6984b964a15a5498ce99dfd41f5" is_pre_election: true
I20260924 23:06:42.434998 7667 raft_consensus.cc:2454] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b487ca706f14b5dabeb532009e2b2ed in term 0.
I20260924 23:06:42.435040 7531 raft_consensus.cc:2454] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b487ca706f14b5dabeb532009e2b2ed in term 0.
I20260924 23:06:42.435189 7329 leader_election.cc:302] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae6392fb8a648d59312a7b169c22d0b, 4b487ca706f14b5dabeb532009e2b2ed; no voters:
I20260924 23:06:42.435338 7822 raft_consensus.cc:2781] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:42.435380 7822 raft_consensus.cc:482] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:42.435410 7822 raft_consensus.cc:3037] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:42.436084 7822 raft_consensus.cc:504] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.436228 7822 leader_election.cc:288] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [CANDIDATE]: Term 1 election: Requested vote from peers 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683), 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601)
I20260924 23:06:42.436371 7531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18ef837a3e04c9f9c5608e6ca0e44a0" candidate_uuid: "4b487ca706f14b5dabeb532009e2b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f6d6984b964a15a5498ce99dfd41f5"
I20260924 23:06:42.436383 7667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18ef837a3e04c9f9c5608e6ca0e44a0" candidate_uuid: "4b487ca706f14b5dabeb532009e2b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae6392fb8a648d59312a7b169c22d0b"
I20260924 23:06:42.436446 7667 raft_consensus.cc:3037] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:42.436446 7531 raft_consensus.cc:3037] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:42.437140 7531 raft_consensus.cc:2454] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b487ca706f14b5dabeb532009e2b2ed in term 1.
I20260924 23:06:42.437271 7667 raft_consensus.cc:2454] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b487ca706f14b5dabeb532009e2b2ed in term 1.
I20260924 23:06:42.437422 7327 leader_election.cc:302] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b487ca706f14b5dabeb532009e2b2ed, 61f6d6984b964a15a5498ce99dfd41f5; no voters:
I20260924 23:06:42.437523 7822 raft_consensus.cc:2781] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:42.437682 7822 raft_consensus.cc:686] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 1 LEADER]: Becoming Leader. State: Replica: 4b487ca706f14b5dabeb532009e2b2ed, State: Running, Role: LEADER
I20260924 23:06:42.437750 7822 consensus_queue.cc:237] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } }
I20260924 23:06:42.438023 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.437915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:44951)
0924 23:06:42.437967 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.437996 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:06:42.438508 7074 catalog_manager.cc:5495] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4b487ca706f14b5dabeb532009e2b2ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b487ca706f14b5dabeb532009e2b2ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:42.455237 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.452904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.0.98.1:37045)
0924 23:06:42.452966 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:42.452968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:42.452975 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:42.453011 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:42.453012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:42.453030 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:42.453094 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:42.453254 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.453701 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.454557 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:42.454736 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:42.454788 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:42.454967 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:42.455064 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:42.455117 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:42.455165 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 23:06:42.455183 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:06:42.456336 7531 raft_consensus.cc:1261] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Refusing update from remote peer 4b487ca706f14b5dabeb532009e2b2ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:42.456415 7667 raft_consensus.cc:1261] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Refusing update from remote peer 4b487ca706f14b5dabeb532009e2b2ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:42.456626 7822 consensus_queue.cc:1035] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f6d6984b964a15a5498ce99dfd41f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:42.456840 7822 consensus_queue.cc:1035] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae6392fb8a648d59312a7b169c22d0b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:42.457428 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.457327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:38909)
0924 23:06:42.457367 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.457396 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 23:06:42.458071 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.457831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:45981)
0924 23:06:42.457998 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.458033 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260924 23:06:42.458101 7829 mvcc.cc:205] Tried to move back new op lower bound from 7333032765258989568 to 7333032765185765376. Current Snapshot: MvccSnapshot[applied={T|T < 7333032765258989568}]
I20260924 23:06:42.458410 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.458318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:42553)
0924 23:06:42.458355 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.458379 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 23:06:42.459271 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.459027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:49749)
0924 23:06:42.459183 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.459243 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 23:06:42.459475 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.459174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:56433)
0924 23:06:42.459418 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.459446 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260924 23:06:42.460206 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.460118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:47629)
0924 23:06:42.460154 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.460181 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 23:06:42.461005 7074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:42.461092 7074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:42.462306 7074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6386922bf36145bf863f98aeb7dd9de3: Sending DeleteTablet for 3 replicas of tablet e18ef837a3e04c9f9c5608e6ca0e44a0
I20260924 23:06:42.462586 7374 tablet_service.cc:1514] Processing DeleteTablet for tablet e18ef837a3e04c9f9c5608e6ca0e44a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:42 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34014
I20260924 23:06:42.462607 7511 tablet_service.cc:1514] Processing DeleteTablet for tablet e18ef837a3e04c9f9c5608e6ca0e44a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:42 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40642
I20260924 23:06:42.462648 7647 tablet_service.cc:1514] Processing DeleteTablet for tablet e18ef837a3e04c9f9c5608e6ca0e44a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:42 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:52660
I20260924 23:06:42.462802 7838 tablet_replica.cc:330] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: stopping tablet replica
I20260924 23:06:42.462858 7838 raft_consensus.cc:2229] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:42.462905 7838 raft_consensus.cc:2258] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:42.463014 7839 tablet_replica.cc:330] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed: stopping tablet replica
I20260924 23:06:42.463061 7839 raft_consensus.cc:2229] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:42.463136 7839 raft_consensus.cc:2258] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4b487ca706f14b5dabeb532009e2b2ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:42.463133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 7221
I20260924 23:06:42.463238 7838 ts_tablet_manager.cc:1905] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:42.465133 7838 ts_tablet_manager.cc:1918] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:42.465176 7838 log.cc:1196] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/e18ef837a3e04c9f9c5608e6ca0e44a0
I20260924 23:06:42.465397 7838 ts_tablet_manager.cc:1939] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 61f6d6984b964a15a5498ce99dfd41f5: Deleting consensus metadata
I20260924 23:06:42.465689 7058 catalog_manager.cc:4841] TS 61f6d6984b964a15a5498ce99dfd41f5 (127.0.98.2:37683): tablet e18ef837a3e04c9f9c5608e6ca0e44a0 (table test-table [id=97f50e08ce824982ba45d79d79c5cc79]) successfully deleted
I20260924 23:06:42.469590 7840 tablet_replica.cc:330] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: stopping tablet replica
I20260924 23:06:42.469666 7840 raft_consensus.cc:2229] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:42.469669 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:42.469539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37045 (local address 127.0.0.1:34030)
0924 23:06:42.469591 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:42.469615 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:06:42.469764 7840 raft_consensus.cc:2258] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:42.469808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 7448
W20260924 23:06:42.469816 7059 catalog_manager.cc:4584] TS 4b487ca706f14b5dabeb532009e2b2ed (127.0.98.1:37045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e18ef837a3e04c9f9c5608e6ca0e44a0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37045: connect: Connection refused (error 111)
I20260924 23:06:42.470541 7840 ts_tablet_manager.cc:1905] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:42.472200 7840 ts_tablet_manager.cc:1918] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:42.472280 7840 log.cc:1196] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/e18ef837a3e04c9f9c5608e6ca0e44a0
I20260924 23:06:42.472533 7840 ts_tablet_manager.cc:1939] T e18ef837a3e04c9f9c5608e6ca0e44a0 P 4ae6392fb8a648d59312a7b169c22d0b: Deleting consensus metadata
I20260924 23:06:42.472879 7060 catalog_manager.cc:4841] TS 4ae6392fb8a648d59312a7b169c22d0b (127.0.98.3:46601): tablet e18ef837a3e04c9f9c5608e6ca0e44a0 (table test-table [id=97f50e08ce824982ba45d79d79c5cc79]) successfully deleted
I20260924 23:06:42.474987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 7584
I20260924 23:06:42.481050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 6941
I20260924 23:06:42.485801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 7128
2026-09-24T23:06:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6764 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:42 dist-test-slave-ph9j krb5kdc[7843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:42 dist-test-slave-ph9j krb5kdc[7843](info): set up 2 sockets
Sep 24 23:06:42 dist-test-slave-ph9j krb5kdc[7843](info): commencing operation
krb5kdc: starting...
W20260924 23:06:43.518410 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:43 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 23:06:43.636824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36767
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:43.716112 7856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:43.716334 7856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:43.716372 7856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:43.717504 7856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:43.717571 7856 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 23:06:43.717595 7856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:43.717628 7856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:43.717654 7856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:43.717682 7856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:43.719146 7856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36767
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:43.719408 7856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:43.719622 7856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:43.724805 7864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:43.724781 7861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:43.724812 7856 server_base.cc:1034] running on GCE node
W20260924 23:06:43.724781 7862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:43.725231 7856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:43.725461 7856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:43.726595 7856 hybrid_clock.cc:648] HybridClock initialized: now 1790291203726576 us; error 43 us; skew 500 ppm
Sep 24 23:06:43 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291203, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:43.734520 7856 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 23:06:43.734899 7856 webserver.cc:460] Webserver started at http://127.0.98.62:34365/ using document root <none> and password file <none>
I20260924 23:06:43.735081 7856 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:43.735121 7856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:43.735208 7856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:43.736198 7856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "256755e405974fd98380b77840ad4f37"
format_stamp: "Formatted at 2026-09-24 23:06:43 on dist-test-slave-ph9j"
server_key: "924a315f6df8c7b42262f06a4176d7ac"
server_key_iv: "e59f5693690bb6dc534dbacbbd13d595"
server_key_version: "encryptionkey@0"
I20260924 23:06:43.736490 7856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "256755e405974fd98380b77840ad4f37"
format_stamp: "Formatted at 2026-09-24 23:06:43 on dist-test-slave-ph9j"
server_key: "924a315f6df8c7b42262f06a4176d7ac"
server_key_iv: "e59f5693690bb6dc534dbacbbd13d595"
server_key_version: "encryptionkey@0"
I20260924 23:06:43.737743 7856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:06:43.738546 7871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:43.738732 7856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:43.738806 7856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "256755e405974fd98380b77840ad4f37"
format_stamp: "Formatted at 2026-09-24 23:06:43 on dist-test-slave-ph9j"
server_key: "924a315f6df8c7b42262f06a4176d7ac"
server_key_iv: "e59f5693690bb6dc534dbacbbd13d595"
server_key_version: "encryptionkey@0"
I20260924 23:06:43.738858 7856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:43.762300 7856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:43.764685 7856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:43.764809 7856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:43.769449 7856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36767
I20260924 23:06:43.769487 7933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36767 every 8 connection(s)
I20260924 23:06:43.769814 7856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:43.770462 7934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:43.772732 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Bootstrap starting.
I20260924 23:06:43.773336 7934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:43.773703 7934 log.cc:824] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:43.773725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 7856
I20260924 23:06:43.773805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:43.774048 392 external_mini_cluster.cc:1428] Setting key b8601b7547d2ed9e0848da406b5cfd86
I20260924 23:06:43.774560 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: No bootstrap required, opened a new log
I20260924 23:06:43.776696 7934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256755e405974fd98380b77840ad4f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36767 } }
I20260924 23:06:43.776866 7934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:43.776906 7934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 256755e405974fd98380b77840ad4f37, State: Initialized, Role: FOLLOWER
I20260924 23:06:43.777081 7934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256755e405974fd98380b77840ad4f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36767 } }
I20260924 23:06:43.777171 7934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:43.777205 7934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:43.777256 7934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:43.778014 7934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256755e405974fd98380b77840ad4f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36767 } }
I20260924 23:06:43.778102 7934 leader_election.cc:302] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 256755e405974fd98380b77840ad4f37; no voters:
I20260924 23:06:43.778280 7934 leader_election.cc:288] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:43.778329 7939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:43.778499 7939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [term 1 LEADER]: Becoming Leader. State: Replica: 256755e405974fd98380b77840ad4f37, State: Running, Role: LEADER
I20260924 23:06:43.778558 7934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:43.778635 7939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256755e405974fd98380b77840ad4f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36767 } }
I20260924 23:06:43.779138 7939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 256755e405974fd98380b77840ad4f37. Latest consensus state: current_term: 1 leader_uuid: "256755e405974fd98380b77840ad4f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256755e405974fd98380b77840ad4f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36767 } } }
I20260924 23:06:43.779294 7939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:43.779541 7948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:43.779150 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "256755e405974fd98380b77840ad4f37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "256755e405974fd98380b77840ad4f37" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36767 } } }
I20260924 23:06:43.779973 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:43.780068 7948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:43.781598 7948 catalog_manager.cc:1347] Generated new cluster ID: 1f37eb308f3842d39523b3ebb5a4f5a4
I20260924 23:06:43.781651 7948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:06:43 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:43.792068 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:43.774571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.0.98.62:36767)
0924 23:06:43.774726 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:43.774740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:43.774768 (+ 28us) server_negotiation.cc:407] Connection header received
0924 23:06:43.776676 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:43.776684 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:43.776735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:43.776883 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:43.777706 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.779081 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.780211 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.780855 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.780967 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:43.788317 (+ 7350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:43.788322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:43.788322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:43.788336 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:43.790385 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:43.790815 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:43.790819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:43.790819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:43.790923 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:43.791135 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:43.791136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:43.791136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:43.791431 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:43.791566 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:43.791701 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 23:06:43.791831 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260924 23:06:43.795379 7948 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:43.795890 7948 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:43.801282 7948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Generated new TSK 0
I20260924 23:06:43.801465 7948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 23:06:43.826978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36767
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:43.908420 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:43.908643 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:43.908679 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:43.909806 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:43.909842 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:43.909860 7960 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:43.909925 7960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:43.911558 7960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:43.911751 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:43.911922 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:43.917092 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:43.917083 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:43.917083 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:43.917152 7960 server_base.cc:1034] running on GCE node
I20260924 23:06:43.917557 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:43.917752 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:43.918887 7960 hybrid_clock.cc:648] HybridClock initialized: now 1790291203918902 us; error 52 us; skew 500 ppm
Sep 24 23:06:43 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291203, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:43.921366 7960 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 23:06:43.921737 7960 webserver.cc:460] Webserver started at http://127.0.98.1:44509/ using document root <none> and password file <none>
I20260924 23:06:43.922000 7960 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:43.922048 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:43.922179 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:43.923295 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "99fbac0020264eaba1c372a735e97af0"
format_stamp: "Formatted at 2026-09-24 23:06:43 on dist-test-slave-ph9j"
server_key: "0f6c120a3739d548810223d02dc21bb0"
server_key_iv: "8f01304c34f7347ba2f0cac39d4f4408"
server_key_version: "encryptionkey@0"
I20260924 23:06:43.923648 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "99fbac0020264eaba1c372a735e97af0"
format_stamp: "Formatted at 2026-09-24 23:06:43 on dist-test-slave-ph9j"
server_key: "0f6c120a3739d548810223d02dc21bb0"
server_key_iv: "8f01304c34f7347ba2f0cac39d4f4408"
server_key_version: "encryptionkey@0"
I20260924 23:06:43.925002 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:43.925797 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:43.926090 7960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:43.926189 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "99fbac0020264eaba1c372a735e97af0"
format_stamp: "Formatted at 2026-09-24 23:06:43 on dist-test-slave-ph9j"
server_key: "0f6c120a3739d548810223d02dc21bb0"
server_key_iv: "8f01304c34f7347ba2f0cac39d4f4408"
server_key_version: "encryptionkey@0"
I20260924 23:06:43.926283 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:43.950495 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:43.952894 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:43.953056 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:43.953564 7960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:43.953630 7960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:43.953678 7960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:43.953703 7960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:43.960417 7960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38585
I20260924 23:06:43.960498 8090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38585 every 8 connection(s)
I20260924 23:06:43.960803 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:06:43 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291203, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:43.961612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 7960
I20260924 23:06:43.961733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:43.961984 392 external_mini_cluster.cc:1428] Setting key 254638201d13ff62ab2809fa07e8319a
I20260924 23:06:43.964663 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:43.954113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46189 (local address 127.0.98.62:36767)
0924 23:06:43.954180 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:43.954183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:43.954421 (+ 238us) server_negotiation.cc:407] Connection header received
0924 23:06:43.954922 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:43.954924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:43.954942 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:43.955033 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:43.955909 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.956777 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.958355 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.958708 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.958788 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:43.961400 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:43.961401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:43.961402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:43.961409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:43.963315 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:43.963690 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:43.963691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:43.963692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:43.963757 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:43.964018 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:43.964019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:43.964019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:43.964141 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:43.964214 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:43.964488 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 23:06:43.964573 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:06:43.964830 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:43.954116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.98.1:46189)
0924 23:06:43.954288 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:43.954328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:43.954827 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:43.955128 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:43.955132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:43.955504 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:43.955799 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:43.955805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.956912 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.956913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:43.958251 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:43.958254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.958794 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.958796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:43.958935 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:43.959497 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:43.959506 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:43.961319 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:43.963396 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:43.963402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:43.963404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:43.963617 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:43.963822 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:43.963822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:43.963822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:43.963941 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:43.964216 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:43.964223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:43.964356 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 23:06:43.964460 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 23:06:43.964607 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260924 23:06:43.965298 8091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36767
I20260924 23:06:43.965384 8091 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:43.965581 8091 heartbeater.cc:511] Master 127.0.98.62:36767 requested a full tablet report, sending...
I20260924 23:06:43.966154 7887 ts_manager.cc:194] Registered new tserver with Master: 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585)
I20260924 23:06:43.966998 7887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46189
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 23:06:43.972532 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:43.966447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.98.1:45491)
0924 23:06:43.966625 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:43.966637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:43.966705 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:43.966840 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:43.966840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:43.967051 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:43.967118 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:43.967120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.968008 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.968009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:43.968884 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:43.968887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.969219 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.969220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:43.969293 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:43.969670 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:43.969675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:43.970091 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:43.971635 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:43.971636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:43.971636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:43.971811 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:43.972018 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:43.972018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:43.972019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:43.972071 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:43.972290 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:43.972291 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:43.972327 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 23:06:43.972385 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:06:43.972459 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260924 23:06:43.972558 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:43.966521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45491 (local address 127.0.98.62:36767)
0924 23:06:43.966564 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:43.966566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:43.966710 (+ 144us) server_negotiation.cc:407] Connection header received
0924 23:06:43.966769 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:43.966770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:43.966779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:43.966826 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:43.967183 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.967896 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.968969 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:43.969142 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:43.969189 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:43.970180 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:43.970181 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:43.970181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:43.970187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:43.971530 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:43.971915 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:43.971916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:43.971916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:43.971962 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:43.972121 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:43.972121 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:43.972122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:43.972210 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:43.972288 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:43.972425 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 23:06:43.972500 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 23:06:43.988272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36767
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:44.067471 8100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:44.067667 8100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:44.067690 8100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:44.068765 8100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:44.068799 8100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:44.068819 8100 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:44.068836 8100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:44.070223 8100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:44.070398 8100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:44.070564 8100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:44.075536 8105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:44.075511 8108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:44.075505 8106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:44.075709 8100 server_base.cc:1034] running on GCE node
I20260924 23:06:44.075893 8100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:44.076095 8100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:44.077229 8100 hybrid_clock.cc:648] HybridClock initialized: now 1790291204077218 us; error 26 us; skew 500 ppm
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:44.079820 8100 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 23:06:44.080289 8100 webserver.cc:460] Webserver started at http://127.0.98.2:34949/ using document root <none> and password file <none>
I20260924 23:06:44.080528 8100 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:44.080577 8100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:44.080665 8100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:44.081947 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "6dee32522c2547c49b683f876c047461"
format_stamp: "Formatted at 2026-09-24 23:06:44 on dist-test-slave-ph9j"
server_key: "4fde8fa313138a3e06591a40f04e3787"
server_key_iv: "9b81609e5b24e5318f64960821f49389"
server_key_version: "encryptionkey@0"
I20260924 23:06:44.082342 8100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6dee32522c2547c49b683f876c047461"
format_stamp: "Formatted at 2026-09-24 23:06:44 on dist-test-slave-ph9j"
server_key: "4fde8fa313138a3e06591a40f04e3787"
server_key_iv: "9b81609e5b24e5318f64960821f49389"
server_key_version: "encryptionkey@0"
I20260924 23:06:44.083552 8100 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:44.084259 8115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.084465 8100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:44.084537 8100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "6dee32522c2547c49b683f876c047461"
format_stamp: "Formatted at 2026-09-24 23:06:44 on dist-test-slave-ph9j"
server_key: "4fde8fa313138a3e06591a40f04e3787"
server_key_iv: "9b81609e5b24e5318f64960821f49389"
server_key_version: "encryptionkey@0"
I20260924 23:06:44.084587 8100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:44.097114 8100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:44.098955 8100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:44.099068 8100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:44.099557 8100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:44.099597 8100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.099619 8100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:44.099632 8100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:44.106899 8100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34843
I20260924 23:06:44.106989 8230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34843 every 8 connection(s)
I20260924 23:06:44.107297 8100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:44.108804 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.099914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59575 (local address 127.0.98.62:36767)
0924 23:06:44.099976 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.099979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.100225 (+ 246us) server_negotiation.cc:407] Connection header received
0924 23:06:44.100746 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.100747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.100759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.100816 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.101397 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.101985 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.103035 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.103242 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.103317 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.105971 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.105980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.105981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.105989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.107533 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.107979 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.107981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.107982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.108055 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.108300 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.108301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.108301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.108403 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.108454 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.108674 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.108740 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:06:44.109004 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.099917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.98.2:59575)
0924 23:06:44.100095 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.100133 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.100636 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.100890 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.100893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.101122 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:44.101294 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.101298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.102086 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.102087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.102928 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.102930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.103322 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.103323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.103436 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.104027 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:44.104035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:44.105853 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:44.107668 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.107672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.107673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.107882 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.108140 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.108141 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.108141 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.108243 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.108459 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:44.108463 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:44.108570 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.108648 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.108760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260924 23:06:44.109508 8231 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36767
I20260924 23:06:44.109580 8231 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:44.109748 8231 heartbeater.cc:511] Master 127.0.98.62:36767 requested a full tablet report, sending...
I20260924 23:06:44.110147 7887 ts_manager.cc:194] Registered new tserver with Master: 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843)
I20260924 23:06:44.110631 7887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59575
I20260924 23:06:44.112582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 8100
I20260924 23:06:44.112701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:44.112944 392 external_mini_cluster.cc:1428] Setting key 65f4a5893939a0142c73306ada641dad
I20260924 23:06:44.115952 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.110602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56167 (local address 127.0.98.62:36767)
0924 23:06:44.110646 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.110648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.110739 (+ 91us) server_negotiation.cc:407] Connection header received
0924 23:06:44.110773 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.110774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.110784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.110838 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.111160 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.111710 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.112343 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.112515 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.112561 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.113499 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.113500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.113500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.113507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.114828 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.115170 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.115171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.115171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.115222 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.115432 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.115433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.115433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.115568 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.115646 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.115791 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.115864 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:44.115971 8237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.110532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.98.2:56167)
0924 23:06:44.110657 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.110668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.110733 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.110839 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.110840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.111031 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:44.111087 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.111089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.111791 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.111792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.112278 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.112279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.112584 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.112585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.112660 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.113027 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:44.113032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:44.113426 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:44.114914 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.114916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.114916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.115085 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.115295 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.115296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.115297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.115353 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.115638 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:44.115639 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:44.115681 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.115769 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.115864 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 23:06:44.139213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36767
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:44.220916 8240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:44.221129 8240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:44.221154 8240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:44.222265 8240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:44.222301 8240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:44.222321 8240 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 23:06:44.222338 8240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:44.223984 8240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:44.224165 8240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:44.224335 8240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:06:44.229334 8240 server_base.cc:1034] running on GCE node
W20260924 23:06:44.229285 8246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:44.229357 8245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:44.229286 8248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:44.229681 8240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:44.229980 8240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:44.231141 8240 hybrid_clock.cc:648] HybridClock initialized: now 1790291204231126 us; error 27 us; skew 500 ppm
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:44.233546 8240 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 23:06:44.234045 8240 webserver.cc:460] Webserver started at http://127.0.98.3:40671/ using document root <none> and password file <none>
I20260924 23:06:44.234270 8240 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:44.234337 8240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:44.234472 8240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:44.235530 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "d6aac5116b60467eab173ab8887d8d6b"
format_stamp: "Formatted at 2026-09-24 23:06:44 on dist-test-slave-ph9j"
server_key: "3f904d7d72a34f6e7fb9b602bc33825d"
server_key_iv: "d98899d3605c687bb7f239d27a2e9863"
server_key_version: "encryptionkey@0"
I20260924 23:06:44.235903 8240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d6aac5116b60467eab173ab8887d8d6b"
format_stamp: "Formatted at 2026-09-24 23:06:44 on dist-test-slave-ph9j"
server_key: "3f904d7d72a34f6e7fb9b602bc33825d"
server_key_iv: "d98899d3605c687bb7f239d27a2e9863"
server_key_version: "encryptionkey@0"
I20260924 23:06:44.237231 8240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:06:44.237984 8255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.238164 8240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:44.238262 8240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "d6aac5116b60467eab173ab8887d8d6b"
format_stamp: "Formatted at 2026-09-24 23:06:44 on dist-test-slave-ph9j"
server_key: "3f904d7d72a34f6e7fb9b602bc33825d"
server_key_iv: "d98899d3605c687bb7f239d27a2e9863"
server_key_version: "encryptionkey@0"
I20260924 23:06:44.238348 8240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:44.252226 8240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:44.254016 8240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:44.254176 8240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:44.254666 8240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:44.254733 8240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.254777 8240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:44.254803 8240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.261690 8240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40025
I20260924 23:06:44.261750 8370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40025 every 8 connection(s)
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:44.262115 8240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:44.264459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 8240
I20260924 23:06:44.264598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:44.264874 392 external_mini_cluster.cc:1428] Setting key 15ba67575889654455939c289619a877
I20260924 23:06:44.265743 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.255193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36089 (local address 127.0.98.62:36767)
0924 23:06:44.255240 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.255243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.255442 (+ 199us) server_negotiation.cc:407] Connection header received
0924 23:06:44.255967 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.255968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.255980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.256044 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.256976 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.257787 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.259512 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.259826 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.259886 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.262603 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.262605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.262605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.262613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.264029 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.264577 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.264578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.264579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.264670 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.265014 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.265015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.265016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.265168 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.265233 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.265557 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.265642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:06:44.265995 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.255145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.98.3:36089)
0924 23:06:44.255311 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.255354 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.255857 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.256159 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.256164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.256561 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:44.256860 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.256868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.257936 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.257937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.259408 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.259412 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.259912 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.259913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.260085 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.260699 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:44.260710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:44.262520 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:44.264150 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.264156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.264158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.264474 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.264747 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.264748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.264748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.264898 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.265224 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:44.265229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:44.265425 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.265537 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.265704 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260924 23:06:44.266463 8371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36767
I20260924 23:06:44.266564 8371 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:44.266741 8371 heartbeater.cc:511] Master 127.0.98.62:36767 requested a full tablet report, sending...
I20260924 23:06:44.267128 7887 ts_manager.cc:194] Registered new tserver with Master: d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.267633 7887 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36089
I20260924 23:06:44.272652 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.267491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41671 (local address 127.0.98.62:36767)
0924 23:06:44.267534 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.267535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.267591 (+ 56us) server_negotiation.cc:407] Connection header received
0924 23:06:44.267663 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.267664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.267673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.267727 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.268050 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.268510 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.269194 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.269390 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.269473 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.270394 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.270396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.270396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.270402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.271549 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.271909 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.271910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.271911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.271965 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.272157 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.272158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.272158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.272264 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.272339 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.272491 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.272562 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:06:44.272676 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.267416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.98.3:41671)
0924 23:06:44.267538 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.267549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.267607 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.267732 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.267733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.267919 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:44.267964 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.267966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.268601 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.268602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.269112 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.269114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.269476 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.269477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.269540 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.269997 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:44.270001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:44.270287 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:44.271638 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.271639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.271639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.271829 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.272030 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.272031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.272031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.272092 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.272332 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:44.272333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:44.272375 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.272466 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.272563 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260924 23:06:44.276520 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:06:44.297037 7938 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:44.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.0.98.62:36767)
0924 23:06:44.292084 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.292087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.292099 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:44.292133 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.292134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.292147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.292217 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.292736 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.293226 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.293981 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.294159 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.294206 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.296975 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35912: BlockingRecv error: recv got EOF from 127.0.0.1:35912 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:44.304270 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.297860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.0.98.62:36767)
0924 23:06:44.297945 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.297947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.298012 (+ 65us) server_negotiation.cc:407] Connection header received
0924 23:06:44.298048 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.298048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.298056 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.298099 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.298563 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.299068 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.299732 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.299906 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.299963 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.302285 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.302287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.302287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.302293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.303436 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.303725 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.303725 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.303726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.303772 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.303915 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.303915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.303915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.304010 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.304076 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.304156 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.304207 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 23:06:44.305747 7887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:44.307353 7887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:44.313585 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.309593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.0.1:33904)
0924 23:06:44.309729 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.309774 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.309844 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.310292 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.310295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.310298 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.310374 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.310377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.311373 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.311374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.312515 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.312517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.313268 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.313269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.313345 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.313359 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.313449 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.313508 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260924 23:06:44.313843 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.309774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38585 (local address 127.0.0.1:53728)
0924 23:06:44.310190 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.310200 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.310252 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.310499 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.310500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.310502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.310544 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.310546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.311868 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.311869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.312628 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.312630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.313614 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.313615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.313685 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.313688 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.313734 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.313771 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":59,"threads_started":1}
I20260924 23:06:44.313952 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.309809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.0.98.3:40025)
0924 23:06:44.310046 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.310053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.310065 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:44.310104 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.310106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.310122 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.310529 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.310589 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.311285 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.312609 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.313163 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.313265 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.313347 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.313777 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.313820 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":105,"threads_started":1}
I20260924 23:06:44.314200 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.309997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.0.98.1:38585)
0924 23:06:44.310155 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.310159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.310303 (+ 144us) server_negotiation.cc:407] Connection header received
0924 23:06:44.310342 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.310344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.310361 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.310439 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.310659 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.311783 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.312712 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.313440 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.313513 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.313581 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.314046 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.314085 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260924 23:06:44.314527 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.309594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34843 (local address 127.0.0.1:36414)
0924 23:06:44.309863 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.309919 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.309961 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.310539 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.310540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.310541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.310571 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.310572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.311848 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.311849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.313241 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.313244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.314228 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.314229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.314294 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.314298 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.314329 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.314463 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":29,"thread_start_us":47,"threads_started":1}
I20260924 23:06:44.314625 8304 tablet_service.cc:1467] Processing CreateTablet for tablet 749a01c5f66241b89cda4c377f45ab02 (DEFAULT_TABLE table=test-table [id=9c8b04e7660b4b5dbd446cef6c138651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.314810 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 749a01c5f66241b89cda4c377f45ab02 (DEFAULT_TABLE table=test-table [id=9c8b04e7660b4b5dbd446cef6c138651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.315047 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.309701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.0.98.2:34843)
0924 23:06:44.310370 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.310374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.310383 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:44.310418 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.310421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.310435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.310535 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.310676 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.311756 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.313337 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.314133 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.314793 (+ 660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.314878 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.314904 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.314941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":83,"threads_started":1}
I20260924 23:06:44.315043 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749a01c5f66241b89cda4c377f45ab02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.315156 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749a01c5f66241b89cda4c377f45ab02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.317941 8399 tablet_bootstrap.cc:492] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Bootstrap starting.
I20260924 23:06:44.318322 8400 tablet_bootstrap.cc:492] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Bootstrap starting.
I20260924 23:06:44.318517 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 749a01c5f66241b89cda4c377f45ab02 (DEFAULT_TABLE table=test-table [id=9c8b04e7660b4b5dbd446cef6c138651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.318665 8399 tablet_bootstrap.cc:654] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.318822 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 749a01c5f66241b89cda4c377f45ab02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.318909 8400 tablet_bootstrap.cc:654] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.319041 8399 log.cc:824] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:44.319216 8400 log.cc:824] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:44.319728 8399 tablet_bootstrap.cc:492] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: No bootstrap required, opened a new log
I20260924 23:06:44.319814 8399 ts_tablet_manager.cc:1397] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:06:44.321472 8399 raft_consensus.cc:348] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.321627 8399 raft_consensus.cc:374] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.321671 8399 raft_consensus.cc:729] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.321790 8399 consensus_queue.cc:260] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.322145 8399 ts_tablet_manager.cc:1428] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:06:44.322278 8371 heartbeater.cc:503] Master 127.0.98.62:36767 was elected leader, sending a full tablet report...
I20260924 23:06:44.322506 8400 tablet_bootstrap.cc:492] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: No bootstrap required, opened a new log
I20260924 23:06:44.322582 8400 ts_tablet_manager.cc:1397] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 23:06:44.324230 8400 raft_consensus.cc:348] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.324352 8400 raft_consensus.cc:374] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.324373 8400 raft_consensus.cc:729] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.324519 8400 consensus_queue.cc:260] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.324782 8400 ts_tablet_manager.cc:1428] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:06:44.324797 8091 heartbeater.cc:503] Master 127.0.98.62:36767 was elected leader, sending a full tablet report...
I20260924 23:06:44.322158 8403 tablet_bootstrap.cc:492] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Bootstrap starting.
I20260924 23:06:44.325538 8403 tablet_bootstrap.cc:654] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.325879 8403 log.cc:824] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:44.326560 8403 tablet_bootstrap.cc:492] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: No bootstrap required, opened a new log
I20260924 23:06:44.326642 8403 ts_tablet_manager.cc:1397] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 23:06:44.327955 8403 raft_consensus.cc:348] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.328107 8403 raft_consensus.cc:374] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.328153 8403 raft_consensus.cc:729] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dee32522c2547c49b683f876c047461, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.328272 8403 consensus_queue.cc:260] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.328537 8403 ts_tablet_manager.cc:1428] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:06:44.328598 8231 heartbeater.cc:503] Master 127.0.98.62:36767 was elected leader, sending a full tablet report...
I20260924 23:06:44.346019 8405 raft_consensus.cc:482] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:44.346133 8405 raft_consensus.cc:504] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.346428 8405 leader_election.cc:288] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843), d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.349293 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.346714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35389 (local address 127.0.98.3:40025)
0924 23:06:44.346765 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.346768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.346776 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:44.346804 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.346805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.346816 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.346889 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.347064 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.347595 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.348535 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.348921 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.348974 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.349014 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.349174 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.349215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:06:44.349306 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.346743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51377 (local address 127.0.98.2:34843)
0924 23:06:44.346790 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.346793 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.346943 (+ 150us) server_negotiation.cc:407] Connection header received
0924 23:06:44.346988 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.346989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.347000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.347053 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.347198 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.347692 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.348596 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.348926 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.349010 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.349020 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.349213 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.349248 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:06:44.349303 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.346614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.98.1:35389)
0924 23:06:44.346679 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.346692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.346745 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.346892 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.346893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.346894 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.346972 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.346983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.347674 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.347675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.348444 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.348446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.349023 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.349024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.349105 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.349108 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.349150 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.349212 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 23:06:44.349325 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.346614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34843 (local address 127.0.98.1:51377)
0924 23:06:44.346879 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.346888 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.346944 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.347061 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.347062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.347063 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.347133 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.347135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.347780 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.347781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.348473 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.348475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.349040 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.349041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.349117 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.349120 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.349192 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.349229 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":102,"thread_start_us":110,"threads_started":1}
I20260924 23:06:44.349684 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749a01c5f66241b89cda4c377f45ab02" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aac5116b60467eab173ab8887d8d6b" is_pre_election: true
I20260924 23:06:44.349851 8325 raft_consensus.cc:2454] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 0.
I20260924 23:06:44.349869 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749a01c5f66241b89cda4c377f45ab02" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461" is_pre_election: true
I20260924 23:06:44.350092 8180 raft_consensus.cc:2454] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 0.
I20260924 23:06:44.350107 7976 leader_election.cc:302] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fbac0020264eaba1c372a735e97af0, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.350286 8405 raft_consensus.cc:2781] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:44.350333 8405 raft_consensus.cc:482] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:44.350349 8405 raft_consensus.cc:3037] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.351251 8405 raft_consensus.cc:504] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.351403 8405 leader_election.cc:288] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 election: Requested vote from peers 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843), d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.351490 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749a01c5f66241b89cda4c377f45ab02" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461"
I20260924 23:06:44.351553 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "749a01c5f66241b89cda4c377f45ab02" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aac5116b60467eab173ab8887d8d6b"
I20260924 23:06:44.351598 8180 raft_consensus.cc:3037] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.351636 8325 raft_consensus.cc:3037] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.352276 8180 raft_consensus.cc:2454] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 1.
I20260924 23:06:44.352293 8325 raft_consensus.cc:2454] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 1.
I20260924 23:06:44.352448 7978 leader_election.cc:302] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dee32522c2547c49b683f876c047461, 99fbac0020264eaba1c372a735e97af0; no voters:
I20260924 23:06:44.352545 8405 raft_consensus.cc:2781] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:44.352695 8405 raft_consensus.cc:686] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 1 LEADER]: Becoming Leader. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Running, Role: LEADER
I20260924 23:06:44.352865 8405 consensus_queue.cc:237] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.353718 7888 catalog_manager.cc:5495] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99fbac0020264eaba1c372a735e97af0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "99fbac0020264eaba1c372a735e97af0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } health_report { overall_health: UNKNOWN } } }
W20260924 23:06:44.357816 8232 tablet.cc:2369] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:44.358439 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.356237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.0.98.1:38585)
0924 23:06:44.356299 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.356301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.356308 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:44.356375 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.356376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.356390 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.356441 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:44.356552 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.356966 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.357650 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.357929 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.358018 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.358161 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.358243 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.358301 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.358337 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.358352 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:06:44.359929 8325 raft_consensus.cc:1261] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Refusing update from remote peer 99fbac0020264eaba1c372a735e97af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.359997 8180 raft_consensus.cc:1261] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Refusing update from remote peer 99fbac0020264eaba1c372a735e97af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.360219 8409 consensus_queue.cc:1035] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.360328 8405 consensus_queue.cc:1035] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.362380 8412 mvcc.cc:205] Tried to move back new op lower bound from 7333032773056077824 to 7333032773030088704. Current Snapshot: MvccSnapshot[applied={T|T < 7333032773056077824}]
I20260924 23:06:44.362802 8414 mvcc.cc:205] Tried to move back new op lower bound from 7333032773056077824 to 7333032773030088704. Current Snapshot: MvccSnapshot[applied={T|T < 7333032773056077824}]
I20260924 23:06:44.363505 8413 mvcc.cc:205] Tried to move back new op lower bound from 7333032773056077824 to 7333032773030088704. Current Snapshot: MvccSnapshot[applied={T|T < 7333032773056077824}]
I20260924 23:06:44.364697 7887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:44.364799 7887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:44.365477 7887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Sending DeleteTablet for 3 replicas of tablet 749a01c5f66241b89cda4c377f45ab02
I20260924 23:06:44.365801 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 749a01c5f66241b89cda4c377f45ab02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53728
I20260924 23:06:44.365962 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet 749a01c5f66241b89cda4c377f45ab02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33904
I20260924 23:06:44.366051 8421 tablet_replica.cc:330] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: stopping tablet replica
I20260924 23:06:44.366170 8421 raft_consensus.cc:2229] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:44.366294 8421 raft_consensus.cc:2258] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.366291 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 749a01c5f66241b89cda4c377f45ab02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36414
I20260924 23:06:44.366293 7887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:44.366418 7887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:44.366511 8422 tablet_replica.cc:330] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: stopping tablet replica
I20260924 23:06:44.366600 8422 raft_consensus.cc:2229] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:44.366837 8423 tablet_replica.cc:330] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: stopping tablet replica
I20260924 23:06:44.366927 8423 raft_consensus.cc:2229] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:44.366940 8422 raft_consensus.cc:2258] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.367022 8423 raft_consensus.cc:2258] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.367408 8422 ts_tablet_manager.cc:1905] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.367606 8421 ts_tablet_manager.cc:1905] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.367771 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 07c96571893944269cf5f27f77197264 (DEFAULT_TABLE table=test-table [id=7e35b846e8cf49998f14fa8c2082124a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.367854 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 07c96571893944269cf5f27f77197264 (DEFAULT_TABLE table=test-table [id=7e35b846e8cf49998f14fa8c2082124a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.367887 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c96571893944269cf5f27f77197264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.367960 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c96571893944269cf5f27f77197264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.369184 8400 tablet_bootstrap.cc:492] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Bootstrap starting.
I20260924 23:06:44.369216 8304 tablet_service.cc:1467] Processing CreateTablet for tablet 07c96571893944269cf5f27f77197264 (DEFAULT_TABLE table=test-table [id=7e35b846e8cf49998f14fa8c2082124a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.369313 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c96571893944269cf5f27f77197264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.369465 8421 ts_tablet_manager.cc:1918] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:44.369560 8421 log.cc:1196] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/749a01c5f66241b89cda4c377f45ab02
I20260924 23:06:44.369618 8400 tablet_bootstrap.cc:654] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.369954 8421 ts_tablet_manager.cc:1939] T 749a01c5f66241b89cda4c377f45ab02 P 99fbac0020264eaba1c372a735e97af0: Deleting consensus metadata
I20260924 23:06:44.370452 8423 ts_tablet_manager.cc:1905] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.370620 8399 tablet_bootstrap.cc:492] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Bootstrap starting.
I20260924 23:06:44.370649 8403 tablet_bootstrap.cc:492] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Bootstrap starting.
I20260924 23:06:44.370836 7874 catalog_manager.cc:4841] TS 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585): tablet 749a01c5f66241b89cda4c377f45ab02 (table test-table [id=9c8b04e7660b4b5dbd446cef6c138651]) successfully deleted
I20260924 23:06:44.371035 8399 tablet_bootstrap.cc:654] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.371047 8403 tablet_bootstrap.cc:654] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.371341 8400 tablet_bootstrap.cc:492] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: No bootstrap required, opened a new log
I20260924 23:06:44.371383 8400 ts_tablet_manager.cc:1397] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:06:44.371503 8400 raft_consensus.cc:348] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.371562 8400 raft_consensus.cc:374] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.371582 8400 raft_consensus.cc:729] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.371621 8400 consensus_queue.cc:260] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.371695 8400 ts_tablet_manager.cc:1428] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.372644 8423 ts_tablet_manager.cc:1918] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:44.372730 8423 log.cc:1196] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/749a01c5f66241b89cda4c377f45ab02
I20260924 23:06:44.372929 8422 ts_tablet_manager.cc:1918] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:44.372996 8422 log.cc:1196] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/749a01c5f66241b89cda4c377f45ab02
I20260924 23:06:44.373001 8403 tablet_bootstrap.cc:492] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: No bootstrap required, opened a new log
I20260924 23:06:44.373003 8423 ts_tablet_manager.cc:1939] T 749a01c5f66241b89cda4c377f45ab02 P 6dee32522c2547c49b683f876c047461: Deleting consensus metadata
I20260924 23:06:44.373070 8403 ts_tablet_manager.cc:1397] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:06:44.373212 8403 raft_consensus.cc:348] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.373286 8422 ts_tablet_manager.cc:1939] T 749a01c5f66241b89cda4c377f45ab02 P d6aac5116b60467eab173ab8887d8d6b: Deleting consensus metadata
I20260924 23:06:44.373296 8403 raft_consensus.cc:374] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.373323 8403 raft_consensus.cc:729] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dee32522c2547c49b683f876c047461, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.373369 8403 consensus_queue.cc:260] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.373422 8399 tablet_bootstrap.cc:492] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: No bootstrap required, opened a new log
I20260924 23:06:44.373452 8403 ts_tablet_manager.cc:1428] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.373457 8399 ts_tablet_manager.cc:1397] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 23:06:44.373600 8399 raft_consensus.cc:348] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.373656 8399 raft_consensus.cc:374] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.373646 7872 catalog_manager.cc:4841] TS d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025): tablet 749a01c5f66241b89cda4c377f45ab02 (table test-table [id=9c8b04e7660b4b5dbd446cef6c138651]) successfully deleted
I20260924 23:06:44.373679 8399 raft_consensus.cc:729] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.373728 8399 consensus_queue.cc:260] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.373808 8399 ts_tablet_manager.cc:1428] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.373901 7874 catalog_manager.cc:4841] TS 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843): tablet 749a01c5f66241b89cda4c377f45ab02 (table test-table [id=9c8b04e7660b4b5dbd446cef6c138651]) successfully deleted
I20260924 23:06:44.406080 8407 raft_consensus.cc:482] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:44.406217 8407 raft_consensus.cc:504] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.406595 8407 leader_election.cc:288] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585), d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.410291 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.407019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33499 (local address 127.0.98.3:40025)
0924 23:06:44.407059 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.407061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.407068 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:44.407097 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.407098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.407107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.407179 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.407384 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.407972 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.409316 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.409921 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.409991 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.410002 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.410177 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.410218 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:44.410292 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.406771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.98.2:33499)
0924 23:06:44.406865 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.406877 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.406923 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.407243 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.407244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.407246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.407316 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.407317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.408059 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.408060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.409208 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.409211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.410014 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.410015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.410092 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.410096 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.410150 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.410205 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 23:06:44.410594 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c96571893944269cf5f27f77197264" candidate_uuid: "6dee32522c2547c49b683f876c047461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aac5116b60467eab173ab8887d8d6b" is_pre_election: true
I20260924 23:06:44.410673 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.406900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40283 (local address 127.0.98.1:38585)
0924 23:06:44.406960 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.406962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.406977 (+ 15us) server_negotiation.cc:407] Connection header received
0924 23:06:44.407127 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.407129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.407139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.407169 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.407483 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.408291 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.409594 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.410148 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.410205 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.410214 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.410578 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.410617 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:44.410709 8325 raft_consensus.cc:2454] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dee32522c2547c49b683f876c047461 in term 0.
I20260924 23:06:44.410894 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.406791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38585 (local address 127.0.98.2:40283)
0924 23:06:44.406936 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.406944 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.407009 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.407241 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.407241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.407244 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.407351 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.407353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.408392 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.408393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.409474 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.409482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.410263 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.410264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.410327 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.410330 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.410738 (+ 408us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.410784 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":17,"thread_start_us":66,"threads_started":1}
I20260924 23:06:44.410920 8116 leader_election.cc:302] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dee32522c2547c49b683f876c047461, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.411154 8407 raft_consensus.cc:2781] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:44.411211 8407 raft_consensus.cc:482] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:44.411233 8407 raft_consensus.cc:3037] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.411280 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c96571893944269cf5f27f77197264" candidate_uuid: "6dee32522c2547c49b683f876c047461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fbac0020264eaba1c372a735e97af0" is_pre_election: true
I20260924 23:06:44.411412 8045 raft_consensus.cc:2454] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dee32522c2547c49b683f876c047461 in term 0.
I20260924 23:06:44.412153 8407 raft_consensus.cc:504] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.412295 8407 leader_election.cc:288] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [CANDIDATE]: Term 1 election: Requested vote from peers 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585), d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.412500 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c96571893944269cf5f27f77197264" candidate_uuid: "6dee32522c2547c49b683f876c047461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fbac0020264eaba1c372a735e97af0"
I20260924 23:06:44.412546 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c96571893944269cf5f27f77197264" candidate_uuid: "6dee32522c2547c49b683f876c047461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aac5116b60467eab173ab8887d8d6b"
I20260924 23:06:44.412588 8045 raft_consensus.cc:3037] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.412609 8325 raft_consensus.cc:3037] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.413391 8045 raft_consensus.cc:2454] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dee32522c2547c49b683f876c047461 in term 1.
I20260924 23:06:44.413391 8325 raft_consensus.cc:2454] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dee32522c2547c49b683f876c047461 in term 1.
I20260924 23:06:44.413559 8116 leader_election.cc:302] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dee32522c2547c49b683f876c047461, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.413666 8407 raft_consensus.cc:2781] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:44.413830 8407 raft_consensus.cc:686] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 1 LEADER]: Becoming Leader. State: Replica: 6dee32522c2547c49b683f876c047461, State: Running, Role: LEADER
I20260924 23:06:44.413939 8407 consensus_queue.cc:237] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.414672 7888 catalog_manager.cc:5495] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6dee32522c2547c49b683f876c047461 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6dee32522c2547c49b683f876c047461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } health_report { overall_health: UNKNOWN } } }
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291203, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:44.450155 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.442800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.0.98.62:36767)
0924 23:06:44.442868 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.442870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.443033 (+ 163us) server_negotiation.cc:407] Connection header received
0924 23:06:44.443107 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.443108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.443118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.443164 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.443614 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.444108 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.444817 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.445008 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.445092 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.447352 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.447353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.447354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.447361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.448780 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.449250 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.449251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.449251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.449306 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.449529 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.449530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.449530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.449632 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.449694 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.449953 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.450061 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":42}
I20260924 23:06:44.450157 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.442749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36767 (local address 127.0.0.1:35930)
0924 23:06:44.442945 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.442962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.443050 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.443176 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.443177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.443483 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:44.443528 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.443530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.444190 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.444191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.444731 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.444732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.445091 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.445092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.445139 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.445709 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:44.445719 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:44.447249 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:44.448875 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.448882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.448884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.449154 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.449385 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:44.449386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:44.449386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:44.449461 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:44.449714 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:44.449721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:44.449777 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.449869 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.450012 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":98,"threads_started":1}
I20260924 23:06:44.452450 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35930:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 23:06:44.452689 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:44.454234 8304 tablet_service.cc:1467] Processing CreateTablet for tablet f6c5a10e6b2543c79574a01402a2d472 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e997d6470a2640c5a171053f34e4398e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:44.454288 8025 tablet_service.cc:1467] Processing CreateTablet for tablet f6c5a10e6b2543c79574a01402a2d472 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e997d6470a2640c5a171053f34e4398e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:44.454314 8165 tablet_service.cc:1467] Processing CreateTablet for tablet f6c5a10e6b2543c79574a01402a2d472 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e997d6470a2640c5a171053f34e4398e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 23:06:44.454393 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c5a10e6b2543c79574a01402a2d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.454404 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c5a10e6b2543c79574a01402a2d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.454463 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c5a10e6b2543c79574a01402a2d472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.455834 8399 tablet_bootstrap.cc:492] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: Bootstrap starting.
I20260924 23:06:44.455924 8403 tablet_bootstrap.cc:492] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: Bootstrap starting.
I20260924 23:06:44.456240 8399 tablet_bootstrap.cc:654] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.456310 8403 tablet_bootstrap.cc:654] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.456597 8400 tablet_bootstrap.cc:492] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: Bootstrap starting.
I20260924 23:06:44.456949 8399 tablet_bootstrap.cc:492] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: No bootstrap required, opened a new log
I20260924 23:06:44.457001 8399 ts_tablet_manager.cc:1397] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:44.457145 8399 raft_consensus.cc:348] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.457223 8399 raft_consensus.cc:374] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.457243 8399 raft_consensus.cc:729] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.457212 8400 tablet_bootstrap.cc:654] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.457288 8399 consensus_queue.cc:260] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.457376 8399 ts_tablet_manager.cc:1428] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:44.457527 8404 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.457613 8404 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:44.458069 8400 tablet_bootstrap.cc:492] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: No bootstrap required, opened a new log
I20260924 23:06:44.458133 8400 ts_tablet_manager.cc:1397] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:06:44.458274 8400 raft_consensus.cc:348] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.458360 8400 raft_consensus.cc:374] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.458390 8400 raft_consensus.cc:729] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.458442 8400 consensus_queue.cc:260] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.458532 8400 ts_tablet_manager.cc:1428] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.458559 8405 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.458647 8405 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:44.458938 8403 tablet_bootstrap.cc:492] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: No bootstrap required, opened a new log
I20260924 23:06:44.458989 8403 ts_tablet_manager.cc:1397] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 23:06:44.459277 8403 raft_consensus.cc:348] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.459343 8403 raft_consensus.cc:374] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.459363 8403 raft_consensus.cc:729] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dee32522c2547c49b683f876c047461, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.459414 8403 consensus_queue.cc:260] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.459494 8403 ts_tablet_manager.cc:1428] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:44.459621 8407 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.459681 8407 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 23:06:44.461514 8092 tablet.cc:2369] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:06:44.512722 8372 tablet.cc:2369] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:06:44.619557 8404 raft_consensus.cc:482] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:44.619664 8404 raft_consensus.cc:504] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.620071 8404 leader_election.cc:288] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585), 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843)
I20260924 23:06:44.623420 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.620531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44265 (local address 127.0.98.1:38585)
0924 23:06:44.620588 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.620590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.620601 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:06:44.620635 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.620636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.620649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.620737 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.620938 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.621492 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.622582 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.623072 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.623125 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.623134 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.623332 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.623368 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:06:44.623440 8264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.620312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38585 (local address 127.0.98.3:44265)
0924 23:06:44.620394 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.620407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.620455 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.620770 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.620771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.620773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.620850 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.620852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.621579 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.621580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.622488 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.622491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.623152 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.623153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.623229 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.623233 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.623293 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.623360 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 23:06:44.623879 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.620466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34843 (local address 127.0.98.3:48039)
0924 23:06:44.620667 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.620677 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.620739 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.620937 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.620938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.620939 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:06:44.621013 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.621015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.621664 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.621664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.622860 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.622863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.623517 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.623518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.623670 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.623675 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.623765 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.623802 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260924 23:06:44.623868 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5a10e6b2543c79574a01402a2d472" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fbac0020264eaba1c372a735e97af0" is_pre_election: true
I20260924 23:06:44.623947 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.620606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48039 (local address 127.0.98.2:34843)
0924 23:06:44.620652 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.620653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.620750 (+ 97us) server_negotiation.cc:407] Connection header received
0924 23:06:44.620797 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.620798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.620807 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.620916 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:06:44.621085 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.621579 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.622948 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.623418 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.623690 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.623703 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.623848 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.623892 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:06:44.623965 8045 raft_consensus.cc:2454] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 0.
I20260924 23:06:44.624140 8258 leader_election.cc:302] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fbac0020264eaba1c372a735e97af0, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.624318 8404 raft_consensus.cc:2781] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:44.624383 8404 raft_consensus.cc:482] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:44.624404 8404 raft_consensus.cc:3037] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.624406 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5a10e6b2543c79574a01402a2d472" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461" is_pre_election: true
I20260924 23:06:44.624533 8180 raft_consensus.cc:2454] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 0.
I20260924 23:06:44.625195 8404 raft_consensus.cc:504] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.625298 8404 leader_election.cc:288] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 election: Requested vote from peers 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585), 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843)
I20260924 23:06:44.625473 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5a10e6b2543c79574a01402a2d472" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fbac0020264eaba1c372a735e97af0"
I20260924 23:06:44.625527 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5a10e6b2543c79574a01402a2d472" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461"
I20260924 23:06:44.625566 8045 raft_consensus.cc:3037] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.625592 8180 raft_consensus.cc:3037] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.626375 8045 raft_consensus.cc:2454] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 1.
I20260924 23:06:44.626370 8180 raft_consensus.cc:2454] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 1.
I20260924 23:06:44.626560 8258 leader_election.cc:302] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dee32522c2547c49b683f876c047461, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.626677 8404 raft_consensus.cc:2781] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:44.626845 8404 raft_consensus.cc:686] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [term 1 LEADER]: Becoming Leader. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Running, Role: LEADER
I20260924 23:06:44.626925 8404 consensus_queue.cc:237] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.627298 8440 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: TxnStatusTablet state changed. Reason: New leader d6aac5116b60467eab173ab8887d8d6b. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.627404 8440 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:44.627573 8442 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 23:06:44.627703 8442 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 23:06:44.627717 7888 catalog_manager.cc:5495] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b reported cstate change: term changed from 0 to 1, leader changed from <none> to d6aac5116b60467eab173ab8887d8d6b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } health_report { overall_health: HEALTHY } } }
I20260924 23:06:44.644369 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.641228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.0.98.3:40025)
0924 23:06:44.641272 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.641275 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.641292 (+ 17us) server_negotiation.cc:407] Connection header received
0924 23:06:44.641335 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.641336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.641356 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.641407 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:44.641581 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.642173 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.643446 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.643764 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.643850 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.644057 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.644161 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.644216 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.644292 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.644309 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:44.644421 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.641103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.0.1:33914)
0924 23:06:44.641180 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.641194 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.641252 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.641437 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.641438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.641439 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:44.641509 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.641512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.642250 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.642251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.643344 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.643347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.643870 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.643871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.643950 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.643969 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.644226 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.644233 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.644287 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.644335 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:06:44.645542 8180 raft_consensus.cc:1261] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Refusing update from remote peer d6aac5116b60467eab173ab8887d8d6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.645702 8045 raft_consensus.cc:1261] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Refusing update from remote peer d6aac5116b60467eab173ab8887d8d6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.645992 8440 consensus_queue.cc:1035] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.646171 8404 consensus_queue.cc:1035] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.646615 8407 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: TxnStatusTablet state changed. Reason: New leader d6aac5116b60467eab173ab8887d8d6b. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.646699 8407 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:44.646811 8405 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: TxnStatusTablet state changed. Reason: New leader d6aac5116b60467eab173ab8887d8d6b. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.646893 8405 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:44.647522 8404 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.647609 8404 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:44.647899 8405 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.647977 8405 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P 99fbac0020264eaba1c372a735e97af0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:44.648783 8440 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.648869 8440 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P d6aac5116b60467eab173ab8887d8d6b: This TxnStatusTablet replica's current role is: LEADER
I20260924 23:06:44.649065 8407 tablet_replica.cc:439] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } } }
I20260924 23:06:44.649149 8407 tablet_replica.cc:442] T f6c5a10e6b2543c79574a01402a2d472 P 6dee32522c2547c49b683f876c047461: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 23:06:44.652094 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.649395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.0.98.2:34843)
0924 23:06:44.649431 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.649433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.649440 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:44.649475 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.649476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.649494 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.649558 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:44.649719 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.650482 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.651241 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.651541 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.651610 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.651757 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.651885 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.651957 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.651995 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.652013 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:06:44.657073 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.654570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58077 (local address 127.0.98.3:40025)
0924 23:06:44.654612 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.654614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.654753 (+ 139us) server_negotiation.cc:407] Connection header received
0924 23:06:44.654820 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.654821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.654831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.654878 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:44.655052 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.655569 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.656375 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.656572 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.656662 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.656840 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.656871 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.656917 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.656982 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.657001 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:44.657109 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.654505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.98.2:58077)
0924 23:06:44.654687 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.654696 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.654767 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.654919 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.654920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.654922 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:44.654978 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.654979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.655659 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.655660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.656240 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.656242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.656666 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.656667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.656727 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.656749 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.656916 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.656917 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.656977 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.657020 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260924 23:06:44.659277 8325 raft_consensus.cc:1261] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Refusing update from remote peer 6dee32522c2547c49b683f876c047461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.659283 8045 raft_consensus.cc:1261] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Refusing update from remote peer 6dee32522c2547c49b683f876c047461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.659533 8428 consensus_queue.cc:1035] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.659667 8407 consensus_queue.cc:1035] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.667807 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.664695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34843 (local address 127.0.98.3:34077)
0924 23:06:44.664746 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:44.664759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:44.664798 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:44.664959 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:44.664960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:44.664962 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:44.665018 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.665021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.665958 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.665965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.666877 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:44.666880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.667339 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.667340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:44.667414 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.667429 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.667633 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.667635 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:06:44.667691 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 23:06:44.667732 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 23:06:44.667801 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.664797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34077 (local address 127.0.98.2:34843)
0924 23:06:44.664858 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.664860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.664869 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:44.664890 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.664891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.664900 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.664955 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:44.665092 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.665829 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.666985 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.667269 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.667340 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.667512 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.667545 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.667611 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.667707 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.667728 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:44.677736 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:44.677851 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:44.678608 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Sending DeleteTablet for 3 replicas of tablet 07c96571893944269cf5f27f77197264
I20260924 23:06:44.678898 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c96571893944269cf5f27f77197264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36414
I20260924 23:06:44.679011 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c96571893944269cf5f27f77197264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53728
I20260924 23:06:44.679116 8423 tablet_replica.cc:330] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: stopping tablet replica
I20260924 23:06:44.679173 8423 raft_consensus.cc:2229] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:44.679253 8423 raft_consensus.cc:2258] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.679319 8421 tablet_replica.cc:330] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: stopping tablet replica
I20260924 23:06:44.679369 8421 raft_consensus.cc:2229] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:44.679399 8421 raft_consensus.cc:2258] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.679541 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c96571893944269cf5f27f77197264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33904
I20260924 23:06:44.679629 8422 tablet_replica.cc:330] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: stopping tablet replica
I20260924 23:06:44.679672 8422 raft_consensus.cc:2229] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:44.679700 8422 raft_consensus.cc:2258] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.679757 8421 ts_tablet_manager.cc:1905] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.680191 8423 ts_tablet_manager.cc:1905] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.681535 8421 ts_tablet_manager.cc:1918] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:44.681586 8421 log.cc:1196] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/07c96571893944269cf5f27f77197264
I20260924 23:06:44.681746 8423 ts_tablet_manager.cc:1918] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:44.681794 8423 log.cc:1196] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/07c96571893944269cf5f27f77197264
I20260924 23:06:44.681813 8421 ts_tablet_manager.cc:1939] T 07c96571893944269cf5f27f77197264 P 99fbac0020264eaba1c372a735e97af0: Deleting consensus metadata
I20260924 23:06:44.682070 8423 ts_tablet_manager.cc:1939] T 07c96571893944269cf5f27f77197264 P 6dee32522c2547c49b683f876c047461: Deleting consensus metadata
I20260924 23:06:44.682292 7874 catalog_manager.cc:4841] TS 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585): tablet 07c96571893944269cf5f27f77197264 (table test-table [id=7e35b846e8cf49998f14fa8c2082124a]) successfully deleted
I20260924 23:06:44.685791 7874 catalog_manager.cc:4841] TS 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843): tablet 07c96571893944269cf5f27f77197264 (table test-table [id=7e35b846e8cf49998f14fa8c2082124a]) successfully deleted
I20260924 23:06:44.685894 8422 ts_tablet_manager.cc:1905] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.687538 8422 ts_tablet_manager.cc:1918] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:44.687601 8422 log.cc:1196] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/07c96571893944269cf5f27f77197264
I20260924 23:06:44.687810 8422 ts_tablet_manager.cc:1939] T 07c96571893944269cf5f27f77197264 P d6aac5116b60467eab173ab8887d8d6b: Deleting consensus metadata
I20260924 23:06:44.688267 7872 catalog_manager.cc:4841] TS d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025): tablet 07c96571893944269cf5f27f77197264 (table test-table [id=7e35b846e8cf49998f14fa8c2082124a]) successfully deleted
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:06:44.708344 7938 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:44.703002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.0.98.62:36767)
0924 23:06:44.703053 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.703056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.703201 (+ 145us) server_negotiation.cc:407] Connection header received
0924 23:06:44.703287 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.703288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.703299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.703379 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.703963 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.704452 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.705190 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.705389 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.705475 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.708266 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35946: BlockingRecv error: recv got EOF from 127.0.0.1:35946 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 23:06:44 dist-test-slave-ph9j krb5kdc[7843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 23:06:44.716461 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.709154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.0.98.62:36767)
0924 23:06:44.709192 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.709194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.709463 (+ 269us) server_negotiation.cc:407] Connection header received
0924 23:06:44.709511 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.709512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.709522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.709555 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:44.710208 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.710806 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.711488 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.711682 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.711750 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.713885 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:44.713886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:44.713887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:44.713922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:44.715332 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.715684 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.715686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.715686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.715744 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:44.715946 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:44.715948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:44.715948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:44.716061 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:44.716166 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.716263 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.716344 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:06:44.718093 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:06:44.718286 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:44.719599 8304 tablet_service.cc:1467] Processing CreateTablet for tablet 8677170040a24b1384d93003c10f0a51 (DEFAULT_TABLE table=test-table [id=37ef3f7d915c42308228a331828d2ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.719676 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 8677170040a24b1384d93003c10f0a51 (DEFAULT_TABLE table=test-table [id=37ef3f7d915c42308228a331828d2ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.719731 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8677170040a24b1384d93003c10f0a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.719774 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8677170040a24b1384d93003c10f0a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.721004 8399 tablet_bootstrap.cc:492] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Bootstrap starting.
I20260924 23:06:44.721033 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 8677170040a24b1384d93003c10f0a51 (DEFAULT_TABLE table=test-table [id=37ef3f7d915c42308228a331828d2ed7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.721135 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8677170040a24b1384d93003c10f0a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.721315 8399 tablet_bootstrap.cc:654] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.722163 8399 tablet_bootstrap.cc:492] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: No bootstrap required, opened a new log
I20260924 23:06:44.722201 8399 ts_tablet_manager.cc:1397] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:44.722322 8399 raft_consensus.cc:348] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.722400 8399 raft_consensus.cc:374] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.722427 8399 raft_consensus.cc:729] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.722476 8399 consensus_queue.cc:260] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.722563 8399 ts_tablet_manager.cc:1428] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.722558 8400 tablet_bootstrap.cc:492] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Bootstrap starting.
I20260924 23:06:44.722854 8400 tablet_bootstrap.cc:654] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.723083 8403 tablet_bootstrap.cc:492] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Bootstrap starting.
I20260924 23:06:44.723423 8403 tablet_bootstrap.cc:654] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.724735 8400 tablet_bootstrap.cc:492] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: No bootstrap required, opened a new log
I20260924 23:06:44.724807 8400 ts_tablet_manager.cc:1397] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 23:06:44.724928 8400 raft_consensus.cc:348] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.725014 8400 raft_consensus.cc:374] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.725041 8400 raft_consensus.cc:729] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.725085 8400 consensus_queue.cc:260] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.725180 8400 ts_tablet_manager.cc:1428] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:44.725754 8403 tablet_bootstrap.cc:492] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: No bootstrap required, opened a new log
I20260924 23:06:44.725828 8403 ts_tablet_manager.cc:1397] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 23:06:44.726011 8403 raft_consensus.cc:348] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.726089 8403 raft_consensus.cc:374] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.726112 8403 raft_consensus.cc:729] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dee32522c2547c49b683f876c047461, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.726155 8403 consensus_queue.cc:260] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.726250 8403 ts_tablet_manager.cc:1428] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.793969 8404 raft_consensus.cc:482] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:44.794137 8404 raft_consensus.cc:504] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.794278 8404 leader_election.cc:288] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585), 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843)
I20260924 23:06:44.794468 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677170040a24b1384d93003c10f0a51" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fbac0020264eaba1c372a735e97af0" is_pre_election: true
I20260924 23:06:44.794477 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677170040a24b1384d93003c10f0a51" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461" is_pre_election: true
I20260924 23:06:44.794564 8045 raft_consensus.cc:2454] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 0.
I20260924 23:06:44.794574 8180 raft_consensus.cc:2454] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 0.
I20260924 23:06:44.794711 8258 leader_election.cc:302] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fbac0020264eaba1c372a735e97af0, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.794806 8404 raft_consensus.cc:2781] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:44.794839 8404 raft_consensus.cc:482] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:44.794858 8404 raft_consensus.cc:3037] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.795588 8404 raft_consensus.cc:504] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.795725 8404 leader_election.cc:288] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 election: Requested vote from peers 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585), 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843)
I20260924 23:06:44.795908 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677170040a24b1384d93003c10f0a51" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99fbac0020264eaba1c372a735e97af0"
I20260924 23:06:44.796005 8045 raft_consensus.cc:3037] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.796669 8045 raft_consensus.cc:2454] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 1.
I20260924 23:06:44.796808 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8677170040a24b1384d93003c10f0a51" candidate_uuid: "d6aac5116b60467eab173ab8887d8d6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461"
I20260924 23:06:44.796898 8180 raft_consensus.cc:3037] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.797679 8180 raft_consensus.cc:2454] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6aac5116b60467eab173ab8887d8d6b in term 1.
I20260924 23:06:44.797902 8258 leader_election.cc:302] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6dee32522c2547c49b683f876c047461, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.798035 8404 raft_consensus.cc:2781] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:44.798105 8404 raft_consensus.cc:686] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 1 LEADER]: Becoming Leader. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Running, Role: LEADER
I20260924 23:06:44.798162 8404 consensus_queue.cc:237] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.798820 7888 catalog_manager.cc:5495] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b reported cstate change: term changed from 0 to 1, leader changed from <none> to d6aac5116b60467eab173ab8887d8d6b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d6aac5116b60467eab173ab8887d8d6b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } health_report { overall_health: HEALTHY } } }
I20260924 23:06:44.816524 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:44.814129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.98.3:40025)
0924 23:06:44.814171 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:44.814174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:44.814208 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:06:44.814306 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:44.814307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:44.814319 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:44.814366 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:44.814556 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.815033 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.815808 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:44.816022 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:44.816098 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:44.816256 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:44.816290 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:44.816357 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:44.816417 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 23:06:44.816439 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:06:44.817462 8180 raft_consensus.cc:1261] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Refusing update from remote peer d6aac5116b60467eab173ab8887d8d6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.817462 8045 raft_consensus.cc:1261] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Refusing update from remote peer d6aac5116b60467eab173ab8887d8d6b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:44.817749 8440 consensus_queue.cc:1035] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.817852 8404 consensus_queue.cc:1035] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:44.821857 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:44.821992 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:44.822600 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Sending DeleteTablet for 3 replicas of tablet 8677170040a24b1384d93003c10f0a51
I20260924 23:06:44.822829 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 8677170040a24b1384d93003c10f0a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53728
I20260924 23:06:44.822853 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 8677170040a24b1384d93003c10f0a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36414
I20260924 23:06:44.822921 8421 tablet_replica.cc:330] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: stopping tablet replica
I20260924 23:06:44.822923 8423 tablet_replica.cc:330] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: stopping tablet replica
I20260924 23:06:44.822966 8421 raft_consensus.cc:2229] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:44.822970 8423 raft_consensus.cc:2229] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:44.823010 8421 raft_consensus.cc:2258] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.823014 8423 raft_consensus.cc:2258] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.823357 7888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 23:06:44.823431 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet 8677170040a24b1384d93003c10f0a51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:44 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33904
I20260924 23:06:44.823503 8422 tablet_replica.cc:330] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: stopping tablet replica
W20260924 23:06:44.823508 7888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:06:44.823547 8422 raft_consensus.cc:2229] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:44.823624 8422 raft_consensus.cc:2258] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:44.824113 8422 ts_tablet_manager.cc:1905] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.825644 8025 tablet_service.cc:1467] Processing CreateTablet for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 (DEFAULT_TABLE table=test-table [id=abdc7a334b6640988c448c826571706d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.825675 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 (DEFAULT_TABLE table=test-table [id=abdc7a334b6640988c448c826571706d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.825760 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.825784 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.827291 8403 tablet_bootstrap.cc:492] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Bootstrap starting.
I20260924 23:06:44.827354 8400 tablet_bootstrap.cc:492] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0: Bootstrap starting.
I20260924 23:06:44.827693 8400 tablet_bootstrap.cc:654] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.827696 8403 tablet_bootstrap.cc:654] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.828130 8421 ts_tablet_manager.cc:1905] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.829511 8304 tablet_service.cc:1467] Processing CreateTablet for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 (DEFAULT_TABLE table=test-table [id=abdc7a334b6640988c448c826571706d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:06:44.829836 8403 tablet_bootstrap.cc:492] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: No bootstrap required, opened a new log
I20260924 23:06:44.829902 8403 ts_tablet_manager.cc:1397] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 23:06:44.830063 8403 raft_consensus.cc:348] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.830163 8403 raft_consensus.cc:374] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.830191 8403 raft_consensus.cc:729] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dee32522c2547c49b683f876c047461, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.830230 8403 consensus_queue.cc:260] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.830336 8403 ts_tablet_manager.cc:1428] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:44.830574 8421 ts_tablet_manager.cc:1918] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:44.830636 8421 log.cc:1196] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/8677170040a24b1384d93003c10f0a51
I20260924 23:06:44.830848 8421 ts_tablet_manager.cc:1939] T 8677170040a24b1384d93003c10f0a51 P 99fbac0020264eaba1c372a735e97af0: Deleting consensus metadata
I20260924 23:06:44.831176 7874 catalog_manager.cc:4841] TS 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585): tablet 8677170040a24b1384d93003c10f0a51 (table test-table [id=37ef3f7d915c42308228a331828d2ed7]) successfully deleted
I20260924 23:06:44.831554 8422 ts_tablet_manager.cc:1918] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:44.831555 8304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:44.831629 8422 log.cc:1196] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/8677170040a24b1384d93003c10f0a51
I20260924 23:06:44.831839 8422 ts_tablet_manager.cc:1939] T 8677170040a24b1384d93003c10f0a51 P d6aac5116b60467eab173ab8887d8d6b: Deleting consensus metadata
I20260924 23:06:44.832113 7872 catalog_manager.cc:4841] TS d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025): tablet 8677170040a24b1384d93003c10f0a51 (table test-table [id=37ef3f7d915c42308228a331828d2ed7]) successfully deleted
I20260924 23:06:44.832430 8423 ts_tablet_manager.cc:1905] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:44.833611 8399 tablet_bootstrap.cc:492] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Bootstrap starting.
I20260924 23:06:44.834175 8399 tablet_bootstrap.cc:654] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:44.834856 8399 tablet_bootstrap.cc:492] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: No bootstrap required, opened a new log
I20260924 23:06:44.834903 8399 ts_tablet_manager.cc:1397] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:44.835005 8399 raft_consensus.cc:348] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.835089 8399 raft_consensus.cc:374] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.835116 8399 raft_consensus.cc:729] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6aac5116b60467eab173ab8887d8d6b, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.835160 8399 consensus_queue.cc:260] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.835253 8399 ts_tablet_manager.cc:1428] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:44.835507 8423 ts_tablet_manager.cc:1918] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:06:44.835567 8423 log.cc:1196] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/8677170040a24b1384d93003c10f0a51
I20260924 23:06:44.835775 8423 ts_tablet_manager.cc:1939] T 8677170040a24b1384d93003c10f0a51 P 6dee32522c2547c49b683f876c047461: Deleting consensus metadata
I20260924 23:06:44.837067 7874 catalog_manager.cc:4841] TS 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843): tablet 8677170040a24b1384d93003c10f0a51 (table test-table [id=37ef3f7d915c42308228a331828d2ed7]) successfully deleted
I20260924 23:06:44.837312 8400 tablet_bootstrap.cc:492] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0: No bootstrap required, opened a new log
I20260924 23:06:44.837379 8400 ts_tablet_manager.cc:1397] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260924 23:06:44.837487 8400 raft_consensus.cc:348] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.837574 8400 raft_consensus.cc:374] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:44.837600 8400 raft_consensus.cc:729] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Initialized, Role: FOLLOWER
I20260924 23:06:44.837639 8400 consensus_queue.cc:260] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.837728 8400 ts_tablet_manager.cc:1428] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 23:06:44.977278 8405 raft_consensus.cc:482] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:06:44.977372 8405 raft_consensus.cc:504] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.977608 8405 leader_election.cc:288] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843), d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.977819 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e354d9b0d694c30a4cd89a7fd1f56b5" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aac5116b60467eab173ab8887d8d6b" is_pre_election: true
I20260924 23:06:44.977835 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e354d9b0d694c30a4cd89a7fd1f56b5" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461" is_pre_election: true
I20260924 23:06:44.977963 8325 raft_consensus.cc:2454] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 0.
I20260924 23:06:44.977980 8180 raft_consensus.cc:2454] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 0.
I20260924 23:06:44.978139 7976 leader_election.cc:302] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fbac0020264eaba1c372a735e97af0, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.978245 8405 raft_consensus.cc:2781] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:06:44.978295 8405 raft_consensus.cc:482] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:06:44.978310 8405 raft_consensus.cc:3037] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.979005 8405 raft_consensus.cc:504] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.979189 8405 leader_election.cc:288] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 election: Requested vote from peers 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843), d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025)
I20260924 23:06:44.979314 8325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e354d9b0d694c30a4cd89a7fd1f56b5" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6aac5116b60467eab173ab8887d8d6b"
I20260924 23:06:44.979321 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e354d9b0d694c30a4cd89a7fd1f56b5" candidate_uuid: "99fbac0020264eaba1c372a735e97af0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dee32522c2547c49b683f876c047461"
I20260924 23:06:44.979383 8325 raft_consensus.cc:3037] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.979400 8180 raft_consensus.cc:3037] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:44.980037 8325 raft_consensus.cc:2454] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 1.
I20260924 23:06:44.980039 8180 raft_consensus.cc:2454] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99fbac0020264eaba1c372a735e97af0 in term 1.
I20260924 23:06:44.980201 7976 leader_election.cc:302] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99fbac0020264eaba1c372a735e97af0, d6aac5116b60467eab173ab8887d8d6b; no voters:
I20260924 23:06:44.980309 8405 raft_consensus.cc:2781] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:44.980443 8405 raft_consensus.cc:686] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 1 LEADER]: Becoming Leader. State: Replica: 99fbac0020264eaba1c372a735e97af0, State: Running, Role: LEADER
I20260924 23:06:44.980511 8405 consensus_queue.cc:237] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260924 23:06:44.981113 7888 catalog_manager.cc:5495] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99fbac0020264eaba1c372a735e97af0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "99fbac0020264eaba1c372a735e97af0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99fbac0020264eaba1c372a735e97af0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } health_report { overall_health: UNKNOWN } } }
I20260924 23:06:45.018198 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:45.016233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.0.98.1:38585)
0924 23:06:45.016277 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:45.016279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:45.016288 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:45.016341 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:45.016341 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:45.016349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:45.016399 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:45.016488 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.016940 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.017616 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.017791 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.017841 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:45.018033 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:45.018057 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:45.018094 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:45.018126 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:06:45.018142 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:06:45.022820 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:45.020391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46617 (local address 127.0.98.3:40025)
0924 23:06:45.020471 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:45.020473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:45.020592 (+ 119us) server_negotiation.cc:407] Connection header received
0924 23:06:45.020666 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:45.020666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:45.020676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:45.020731 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:45.020862 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.021311 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.022108 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.022304 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.022377 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:45.022565 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:45.022603 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:45.022671 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:45.022733 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 23:06:45.022750 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:06:45.022842 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:45.020318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.98.1:46617)
0924 23:06:45.020528 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:45.020537 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:45.020607 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:45.020734 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:45.020735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:45.020736 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:45.020776 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:45.020778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.021402 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.021403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:45.022033 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:45.022035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.022378 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.022379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:45.022431 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:45.022451 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:45.022674 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:45.022677 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:06:45.022728 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:06:45.022769 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":116,"threads_started":1}
I20260924 23:06:45.024767 8180 raft_consensus.cc:1261] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Refusing update from remote peer 99fbac0020264eaba1c372a735e97af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:45.024767 8325 raft_consensus.cc:1261] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Refusing update from remote peer 99fbac0020264eaba1c372a735e97af0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:06:45.024946 8479 consensus_queue.cc:1035] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dee32522c2547c49b683f876c047461" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:45.025058 8405 consensus_queue.cc:1035] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6aac5116b60467eab173ab8887d8d6b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:06:45.031320 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:45.029122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42345 (local address 127.0.98.1:38585)
0924 23:06:45.029155 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:45.029157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:45.029164 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:45.029226 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:45.029227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:45.029234 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:45.029292 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:06:45.029397 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.030110 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.030754 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.030923 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.030969 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:45.031088 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:06:45.031109 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:06:45.031140 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:45.031218 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 23:06:45.031241 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:06:45.031356 8377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:45.029068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38585 (local address 127.0.98.3:42345)
0924 23:06:45.029112 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:45.029122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:45.029162 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:45.029300 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:45.029301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:45.029302 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 23:06:45.029342 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:45.029343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.030175 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.030176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:45.030686 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:45.030688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:45.030986 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:45.030987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:45.031037 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:45.031040 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 23:06:45.031162 (+ 122us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 23:06:45.031163 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 23:06:45.031208 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:06:45.031251 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 23:06:45.037623 7888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:45.037714 7888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:06:45.038275 7888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 256755e405974fd98380b77840ad4f37: Sending DeleteTablet for 3 replicas of tablet 0e354d9b0d694c30a4cd89a7fd1f56b5
I20260924 23:06:45.038568 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:45 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53728
I20260924 23:06:45.038618 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:45 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36414
I20260924 23:06:45.038666 8421 tablet_replica.cc:330] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0: stopping tablet replica
I20260924 23:06:45.038697 8423 tablet_replica.cc:330] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: stopping tablet replica
I20260924 23:06:45.038717 8421 raft_consensus.cc:2229] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:06:45.038735 8423 raft_consensus.cc:2229] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:45.038763 8423 raft_consensus.cc:2258] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:45.038767 8421 raft_consensus.cc:2258] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 99fbac0020264eaba1c372a735e97af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:45.038879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 7960
I20260924 23:06:45.038902 8304 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:06:45 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33904
I20260924 23:06:45.039160 8423 ts_tablet_manager.cc:1905] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:45.040330 8422 tablet_replica.cc:330] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: stopping tablet replica
I20260924 23:06:45.040410 8422 raft_consensus.cc:2229] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:06:45.040454 8422 raft_consensus.cc:2258] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:06:45.040820 8422 ts_tablet_manager.cc:1905] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:06:45.041009 8423 ts_tablet_manager.cc:1918] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:45.041069 8423 log.cc:1196] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/0e354d9b0d694c30a4cd89a7fd1f56b5
I20260924 23:06:45.041302 8423 ts_tablet_manager.cc:1939] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P 6dee32522c2547c49b683f876c047461: Deleting consensus metadata
I20260924 23:06:45.041666 7874 catalog_manager.cc:4841] TS 6dee32522c2547c49b683f876c047461 (127.0.98.2:34843): tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 (table test-table [id=abdc7a334b6640988c448c826571706d]) successfully deleted
I20260924 23:06:45.042824 8422 ts_tablet_manager.cc:1918] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 23:06:45.042900 8422 log.cc:1196] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/0e354d9b0d694c30a4cd89a7fd1f56b5
I20260924 23:06:45.043109 8422 ts_tablet_manager.cc:1939] T 0e354d9b0d694c30a4cd89a7fd1f56b5 P d6aac5116b60467eab173ab8887d8d6b: Deleting consensus metadata
I20260924 23:06:45.043480 7872 catalog_manager.cc:4841] TS d6aac5116b60467eab173ab8887d8d6b (127.0.98.3:40025): tablet 0e354d9b0d694c30a4cd89a7fd1f56b5 (table test-table [id=abdc7a334b6640988c448c826571706d]) successfully deleted
W20260924 23:06:45.046696 7874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38585 (error 108)
I20260924 23:06:45.046952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 8100
I20260924 23:06:45.047605 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:45.047039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38585 (local address 127.0.0.1:53764)
0924 23:06:45.047512 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:45.047557 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":41,"threads_started":1}
W20260924 23:06:45.047755 7874 catalog_manager.cc:4584] TS 99fbac0020264eaba1c372a735e97af0 (127.0.98.1:38585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e354d9b0d694c30a4cd89a7fd1f56b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38585: connect: Connection refused (error 111)
I20260924 23:06:45.053617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 8240
I20260924 23:06:45.059996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 7856
2026-09-24T23:06:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2573 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:45 dist-test-slave-ph9j krb5kdc[8491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:45 dist-test-slave-ph9j krb5kdc[8491](info): set up 2 sockets
Sep 24 23:06:45 dist-test-slave-ph9j krb5kdc[8491](info): commencing operation
krb5kdc: starting...
W20260924 23:06:46.094587 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:46 dist-test-slave-ph9j krb5kdc[8491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:46.210088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:46.288682 8504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:46.288902 8504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:46.288924 8504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:46.290149 8504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:46.290194 8504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:46.290208 8504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:46.290220 8504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:46.290233 8504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:46.291759 8504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:46.292011 8504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:46.292250 8504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:46.297026 8509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:46.297057 8512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:46.297021 8510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:46.297242 8504 server_base.cc:1034] running on GCE node
I20260924 23:06:46.297483 8504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:46.297783 8504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:46.298975 8504 hybrid_clock.cc:648] HybridClock initialized: now 1790291206298952 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T23:06:46Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1246 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:06:46 dist-test-slave-ph9j krb5kdc[8517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:06:46 dist-test-slave-ph9j krb5kdc[8517](info): set up 2 sockets
Sep 24 23:06:46 dist-test-slave-ph9j krb5kdc[8517](info): commencing operation
krb5kdc: starting...
W20260924 23:06:47.347225 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:06:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:06:47.473202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:47.554009 8530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:47.554211 8530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:47.554231 8530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:47.555315 8530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:47.555347 8530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:47.555362 8530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:47.555374 8530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:47.555387 8530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:47.556798 8530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:47.556986 8530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:47.557147 8530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:47.561817 8535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:47.561959 8530 server_base.cc:1034] running on GCE node
W20260924 23:06:47.562047 8538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:47.561817 8536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:47.562325 8530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:47.562616 8530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:47.563778 8530 hybrid_clock.cc:648] HybridClock initialized: now 1790291207563768 us; error 50 us; skew 500 ppm
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:47.566242 8530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:47.566649 8530 webserver.cc:460] Webserver started at http://127.0.98.62:36711/ using document root <none> and password file <none>
I20260924 23:06:47.566869 8530 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:47.566936 8530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:47.567095 8530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:47.568122 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "ce0e8e218e6942e893ab33e5bf94e662"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "ee67791ef045f1b711dc3a492d6769a0"
server_key_iv: "c48547b410def5182275c3259f2f9eb5"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.568485 8530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce0e8e218e6942e893ab33e5bf94e662"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "ee67791ef045f1b711dc3a492d6769a0"
server_key_iv: "c48547b410def5182275c3259f2f9eb5"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.569767 8530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:06:47.570580 8545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.570772 8530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:47.570873 8530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "ce0e8e218e6942e893ab33e5bf94e662"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "ee67791ef045f1b711dc3a492d6769a0"
server_key_iv: "c48547b410def5182275c3259f2f9eb5"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.570958 8530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:47.587765 8530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:47.590245 8530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:47.590418 8530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:47.594282 8530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40211
I20260924 23:06:47.594316 8597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40211 every 8 connection(s)
I20260924 23:06:47.594818 8530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:47.595470 8598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:06:47.597720 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Bootstrap starting.
I20260924 23:06:47.597955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 8530
I20260924 23:06:47.598079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:06:47.598285 392 external_mini_cluster.cc:1428] Setting key c44d5334da6fdb9d3bf61063074d438a
I20260924 23:06:47.598489 8598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Neither blocks nor log segments found. Creating new log.
I20260924 23:06:47.598829 8598 log.cc:824] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:47.599493 8598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: No bootstrap required, opened a new log
I20260924 23:06:47.601773 8598 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:47.601961 8598 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:06:47.601995 8598 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0e8e218e6942e893ab33e5bf94e662, State: Initialized, Role: FOLLOWER
I20260924 23:06:47.602110 8598 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:47.602176 8598 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:47.602203 8598 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:47.602249 8598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:06:47.602792 8598 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:47.602921 8598 leader_election.cc:302] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0e8e218e6942e893ab33e5bf94e662; no voters:
I20260924 23:06:47.603089 8598 leader_election.cc:288] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:06:47.603181 8603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:06:47.603318 8603 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 LEADER]: Becoming Leader. State: Replica: ce0e8e218e6942e893ab33e5bf94e662, State: Running, Role: LEADER
I20260924 23:06:47.603346 8598 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:47.603420 8603 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:47.603749 8603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0e8e218e6942e893ab33e5bf94e662. Latest consensus state: current_term: 1 leader_uuid: "ce0e8e218e6942e893ab33e5bf94e662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } } }
I20260924 23:06:47.603757 8604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0e8e218e6942e893ab33e5bf94e662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } } }
I20260924 23:06:47.603842 8603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:47.603848 8604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: This master's current role is: LEADER
W20260924 23:06:47.605291 8618 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:06:47.605342 8618 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:47.605501 8610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:47.605674 8610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:47.607571 8610 catalog_manager.cc:1347] Generated new cluster ID: 3ffc1e593c8d499fbef44132a91a4597
I20260924 23:06:47.607642 8610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:47.610280 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:47.598821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.0.98.62:40211)
0924 23:06:47.599050 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:47.599058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:47.599082 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:06:47.599474 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:47.599480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:47.599532 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:47.599689 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:47.600457 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.601303 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.602387 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.602783 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.602863 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:47.606099 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:47.606103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:47.606104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:47.606115 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:47.608208 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:47.608628 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:47.608631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:47.608631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:47.608741 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:47.608945 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:47.608945 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:47.608946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:47.609238 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:47.609389 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:47.609931 (+ 542us) server_negotiation.cc:299] Negotiation successful
0924 23:06:47.610070 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260924 23:06:47.620852 8610 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:06:47.621387 8610 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:47.628486 8610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Generated new TSK 0
I20260924 23:06:47.628702 8610 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:06:47.656615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40211
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:47.736913 8624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:47.737149 8624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:47.737186 8624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:47.738334 8624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:47.738401 8624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:47.738446 8624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:47.739917 8624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:47.740173 8624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:47.740391 8624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:06:47.745378 8624 server_base.cc:1034] running on GCE node
W20260924 23:06:47.745349 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:47.745450 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:47.745349 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:47.745723 8624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:47.746017 8624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:47.747181 8624 hybrid_clock.cc:648] HybridClock initialized: now 1790291207747164 us; error 32 us; skew 500 ppm
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:47.749641 8624 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:06:47.750135 8624 webserver.cc:460] Webserver started at http://127.0.98.1:34563/ using document root <none> and password file <none>
I20260924 23:06:47.750362 8624 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:47.750429 8624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:47.750564 8624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:47.751667 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "9656943f662a4e92823dfd7bc2434b81"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "6659c63c85ff347c680bb04b79a4869b"
server_key_iv: "d73f415efa5d4b1ecf632ca91a845b0e"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.752043 8624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9656943f662a4e92823dfd7bc2434b81"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "6659c63c85ff347c680bb04b79a4869b"
server_key_iv: "d73f415efa5d4b1ecf632ca91a845b0e"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.753360 8624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:06:47.754108 8639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.754266 8624 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:47.754371 8624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "9656943f662a4e92823dfd7bc2434b81"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "6659c63c85ff347c680bb04b79a4869b"
server_key_iv: "d73f415efa5d4b1ecf632ca91a845b0e"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.754446 8624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:47.762640 8624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:47.764396 8624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:47.764551 8624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:47.764767 8624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:47.765070 8624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:47.765125 8624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.765164 8624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:47.765192 8624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.770853 8624 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42311
I20260924 23:06:47.770952 8752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42311 every 8 connection(s)
I20260924 23:06:47.771252 8624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:06:47.771523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 8624
I20260924 23:06:47.771629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:06:47.771842 392 external_mini_cluster.cc:1428] Setting key 4c73ec16afd51e5642219a61538eacb1
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 23:06:47.782186 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:47.772301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35439 (local address 127.0.98.62:40211)
0924 23:06:47.772343 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:47.772345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:47.772355 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:47.773625 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:47.773626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:47.773637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:47.773697 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:47.774586 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.775363 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.776399 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.776628 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.776707 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:47.779304 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:47.779305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:47.779306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:47.779320 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:47.780821 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:47.781218 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:47.781219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:47.781219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:47.781269 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:47.781537 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:47.781538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:47.781538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:47.781651 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:47.781706 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:47.782003 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 23:06:47.782076 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:06:47.782392 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:47.771988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:35439)
0924 23:06:47.772153 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:47.772193 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:47.773498 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:47.773773 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:47.773778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:47.774216 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:47.774434 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:47.774439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.775469 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.775470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:47.776301 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:47.776304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.776712 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.776713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:47.776851 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:47.777437 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:47.777445 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:47.779055 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:47.780902 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:47.780908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:47.780910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:47.781142 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:47.781343 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:47.781344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:47.781345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:47.781454 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:47.781713 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:47.781717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:47.781840 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 23:06:47.781980 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 23:06:47.782149 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 23:06:47.783003 8753 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40211
I20260924 23:06:47.783120 8753 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:47.783298 8753 heartbeater.cc:511] Master 127.0.98.62:40211 requested a full tablet report, sending...
I20260924 23:06:47.783850 8562 ts_manager.cc:194] Registered new tserver with Master: 9656943f662a4e92823dfd7bc2434b81 (127.0.98.1:42311)
I20260924 23:06:47.784677 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35439
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:06:47.797395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40211
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:47.878441 8758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:47.878643 8758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:47.878667 8758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:47.879734 8758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:47.879767 8758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:47.879793 8758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:47.881197 8758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:47.881371 8758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:47.881597 8758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:47.886523 8764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:47.886528 8766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:47.886528 8763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:47.886785 8758 server_base.cc:1034] running on GCE node
I20260924 23:06:47.887032 8758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:47.887275 8758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:47.888411 8758 hybrid_clock.cc:648] HybridClock initialized: now 1790291207888392 us; error 36 us; skew 500 ppm
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:47.890906 8758 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:06:47.891324 8758 webserver.cc:460] Webserver started at http://127.0.98.2:46397/ using document root <none> and password file <none>
I20260924 23:06:47.891536 8758 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:47.891601 8758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:47.891747 8758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:47.892750 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "eec3513666f541ff8983395934f86ef0"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "286e99ce690fd1df2987e07bcefbc690"
server_key_iv: "3bb4c126d78d41fa651ad14f82b88774"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.893124 8758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eec3513666f541ff8983395934f86ef0"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "286e99ce690fd1df2987e07bcefbc690"
server_key_iv: "3bb4c126d78d41fa651ad14f82b88774"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.894461 8758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:06:47.895174 8773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.895361 8758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.895445 8758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "eec3513666f541ff8983395934f86ef0"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "286e99ce690fd1df2987e07bcefbc690"
server_key_iv: "3bb4c126d78d41fa651ad14f82b88774"
server_key_version: "encryptionkey@0"
I20260924 23:06:47.895505 8758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:47.930944 8758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:47.933317 8758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:47.933430 8758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:47.933619 8758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:47.933990 8758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:47.934031 8758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.934072 8758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:47.934101 8758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:47.939675 8758 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44061
I20260924 23:06:47.939769 8886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44061 every 8 connection(s)
I20260924 23:06:47.940057 8758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:47.942338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 8758
I20260924 23:06:47.942458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:06:47.942667 392 external_mini_cluster.cc:1428] Setting key 0244b3e44325fbf503adca51e4d1ecba
Sep 24 23:06:47 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:47.950132 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:47.940790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42411 (local address 127.0.98.62:40211)
0924 23:06:47.940851 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:47.940853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:47.941066 (+ 213us) server_negotiation.cc:407] Connection header received
0924 23:06:47.942128 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:47.942129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:47.942142 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:47.942196 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:47.942957 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.943537 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.944645 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.944830 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.944881 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:47.947265 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:47.947267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:47.947267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:47.947274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:47.948703 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:47.949176 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:47.949177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:47.949177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:47.949225 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:47.949510 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:47.949511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:47.949511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:47.949605 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:47.949663 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:47.950008 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 23:06:47.950068 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:06:47.950407 8889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:47.940765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:42411)
0924 23:06:47.940933 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:47.940974 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:47.942004 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:47.942275 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:47.942279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:47.942622 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:47.942865 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:47.942870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.943627 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.943628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:47.944544 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:47.944547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:47.944907 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:47.944908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:47.945061 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:47.945524 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:47.945534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:47.947168 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:47.948788 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:47.948795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:47.948798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:47.949097 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:47.949289 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:47.949290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:47.949290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:47.949430 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:47.949672 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:47.949679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:47.949842 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 23:06:47.949982 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 23:06:47.950148 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260924 23:06:47.950909 8887 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40211
I20260924 23:06:47.951002 8887 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 23:06:47.951184 8887 heartbeater.cc:511] Master 127.0.98.62:40211 requested a full tablet report, sending...
I20260924 23:06:47.951547 8562 ts_manager.cc:194] Registered new tserver with Master: eec3513666f541ff8983395934f86ef0 (127.0.98.2:44061)
I20260924 23:06:47.951916 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:06:47.968483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40211
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:48.047396 8892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:48.047595 8892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:48.047663 8892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:48.049254 8892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:48.049299 8892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:48.049326 8892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:48.050904 8892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:48.051155 8892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:48.051378 8892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:48.056279 8900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.056476 8892 server_base.cc:1034] running on GCE node
W20260924 23:06:48.056279 8897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.056392 8898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.056775 8892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:48.057022 8892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:48.058178 8892 hybrid_clock.cc:648] HybridClock initialized: now 1790291208058168 us; error 50 us; skew 500 ppm
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:48.060557 8892 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:06:48.060926 8892 webserver.cc:460] Webserver started at http://127.0.98.3:33465/ using document root <none> and password file <none>
I20260924 23:06:48.061115 8892 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:48.061157 8892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:48.061241 8892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:06:48.062249 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "9404e0326c164e0683a8fb93a6d323ba"
format_stamp: "Formatted at 2026-09-24 23:06:48 on dist-test-slave-ph9j"
server_key: "9f2f18e437121d808f603903ef19135d"
server_key_iv: "03e79a7e6d40d8f33c8dd8a43f674609"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.062595 8892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9404e0326c164e0683a8fb93a6d323ba"
format_stamp: "Formatted at 2026-09-24 23:06:48 on dist-test-slave-ph9j"
server_key: "9f2f18e437121d808f603903ef19135d"
server_key_iv: "03e79a7e6d40d8f33c8dd8a43f674609"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.063772 8892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:48.064397 8907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.064595 8892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:48.064677 8892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "9404e0326c164e0683a8fb93a6d323ba"
format_stamp: "Formatted at 2026-09-24 23:06:48 on dist-test-slave-ph9j"
server_key: "9f2f18e437121d808f603903ef19135d"
server_key_iv: "03e79a7e6d40d8f33c8dd8a43f674609"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.064728 8892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:48.098737 8892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:48.100653 8892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:48.100762 8892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:48.100945 8892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:48.101205 8892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:48.101235 8892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.101258 8892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:48.101271 8892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.107224 8892 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38357
I20260924 23:06:48.107266 9020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38357 every 8 connection(s)
I20260924 23:06:48.107589 8892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:06:48.113178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 8892
I20260924 23:06:48.113301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:06:48.113476 392 external_mini_cluster.cc:1428] Setting key b50532ce1d3837aaa54a1329c5333977
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:06:48.116719 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.108276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36883 (local address 127.0.98.62:40211)
0924 23:06:48.108342 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.108345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.108553 (+ 208us) server_negotiation.cc:407] Connection header received
0924 23:06:48.109524 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.109525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.109538 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.109603 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.110300 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.110801 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.111802 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.111997 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.112083 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.114436 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:48.114437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:48.114437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:48.114461 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:48.115622 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:48.115971 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:48.115972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:48.115972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:48.116023 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:48.116228 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:48.116229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:48.116229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:48.116327 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:48.116376 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:48.116594 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 23:06:48.116655 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:06:48.116885 9023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.108274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:36883)
0924 23:06:48.108435 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.108469 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.109408 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.109673 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.109677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.110014 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.110204 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.110209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.110895 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.110896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.111708 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.111710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.112090 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.112091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.112216 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.112658 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.112666 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.114342 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:06:48.115690 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:48.115695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:48.115697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:48.115905 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:48.116080 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:06:48.116080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:06:48.116081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:06:48.116174 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:06:48.116380 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:06:48.116384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:06:48.116493 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 23:06:48.116575 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:06:48.116690 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260924 23:06:48.117347 9021 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40211
I20260924 23:06:48.117460 9021 heartbeater.cc:464] Registering TS with master...
I20260924 23:06:48.117637 9021 heartbeater.cc:511] Master 127.0.98.62:40211 requested a full tablet report, sending...
I20260924 23:06:48.118042 8562 ts_manager.cc:194] Registered new tserver with Master: 9404e0326c164e0683a8fb93a6d323ba (127.0.98.3:38357)
I20260924 23:06:48.118407 8562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36883
I20260924 23:06:48.125244 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 23:06:48.194765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 8624
I20260924 23:06:48.199044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 8758
I20260924 23:06:48.203234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 8892
I20260924 23:06:48.207465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 8530
I20260924 23:06:48.212213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40211
--webserver_interface=127.0.98.62
--webserver_port=36711
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:06:48.291275 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:48.291524 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:48.291560 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:48.292698 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:06:48.292769 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:48.292793 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:48.292815 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:06:48.292881 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:06:48.294400 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:48.294679 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:48.294912 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:06:48.299697 9030 server_base.cc:1034] running on GCE node
W20260924 23:06:48.299690 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.299688 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.299815 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.300127 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:48.300360 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:48.301501 9030 hybrid_clock.cc:648] HybridClock initialized: now 1790291208301475 us; error 37 us; skew 500 ppm
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:48.303907 9030 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:06:48.304311 9030 webserver.cc:460] Webserver started at http://127.0.98.62:36711/ using document root <none> and password file <none>
I20260924 23:06:48.304546 9030 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:48.304628 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:48.305984 9030 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:48.306614 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.306958 9030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:48.307077 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "ce0e8e218e6942e893ab33e5bf94e662"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "ee67791ef045f1b711dc3a492d6769a0"
server_key_iv: "c48547b410def5182275c3259f2f9eb5"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.307448 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:48.322427 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:48.324167 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:48.324324 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:48.328248 9030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40211
I20260924 23:06:48.328279 9097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40211 every 8 connection(s)
I20260924 23:06:48.328779 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:06:48.330044 9098 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:06:48.330632 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Bootstrap starting.
I20260924 23:06:48.336565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9030
I20260924 23:06:48.336887 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42311
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40211
--webserver_port=34563
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:06:48.339176 9098 log.cc:824] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Log is configured to *not* fsync() on all Append() calls
I20260924 23:06:48.340853 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:06:48.341164 9098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Bootstrap complete.
I20260924 23:06:48.343436 9098 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:48.343655 9098 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0e8e218e6942e893ab33e5bf94e662, State: Initialized, Role: FOLLOWER
I20260924 23:06:48.343824 9098 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:48.343892 9098 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:06:48.343952 9098 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:06:48.344003 9098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:06:48.345001 9098 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:48.345094 9098 leader_election.cc:302] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0e8e218e6942e893ab33e5bf94e662; no voters:
I20260924 23:06:48.345293 9098 leader_election.cc:288] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:06:48.345341 9102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:06:48.345454 9102 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [term 2 LEADER]: Becoming Leader. State: Replica: ce0e8e218e6942e893ab33e5bf94e662, State: Running, Role: LEADER
I20260924 23:06:48.345563 9098 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:06:48.345551 9102 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } }
I20260924 23:06:48.345808 9102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ce0e8e218e6942e893ab33e5bf94e662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } } }
I20260924 23:06:48.345899 9102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:48.346022 9102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0e8e218e6942e893ab33e5bf94e662. Latest consensus state: current_term: 2 leader_uuid: "ce0e8e218e6942e893ab33e5bf94e662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e8e218e6942e893ab33e5bf94e662" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40211 } } }
I20260924 23:06:48.346092 9102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662 [sys.catalog]: This master's current role is: LEADER
I20260924 23:06:48.346411 9106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:06:48.346920 9106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:06:48.347188 9106 catalog_manager.cc:1259] Loaded cluster ID: 3ffc1e593c8d499fbef44132a91a4597
I20260924 23:06:48.347301 9106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:06:48.348098 9106 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:06:48.348281 9106 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce0e8e218e6942e893ab33e5bf94e662: Loaded TSK: 0
I20260924 23:06:48.348490 9106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:06:48.431111 9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:48.431305 9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:48.431327 9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:48.432587 9100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:48.432631 9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:48.432658 9100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:06:48.434162 9100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34563
--tserver_master_addrs=127.0.98.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:48.434378 9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:48.434597 9100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:48.439416 9124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.439427 9125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.439428 9127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.439492 9100 server_base.cc:1034] running on GCE node
I20260924 23:06:48.439813 9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:48.440106 9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:48.441272 9100 hybrid_clock.cc:648] HybridClock initialized: now 1790291208441253 us; error 33 us; skew 500 ppm
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291208, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:48.443897 9100 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 23:06:48.444396 9100 webserver.cc:460] Webserver started at http://127.0.98.1:34563/ using document root <none> and password file <none>
I20260924 23:06:48.444639 9100 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:48.444720 9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:48.446312 9100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:06:48.447016 9134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.447233 9100 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:06:48.447335 9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "9656943f662a4e92823dfd7bc2434b81"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "6659c63c85ff347c680bb04b79a4869b"
server_key_iv: "d73f415efa5d4b1ecf632ca91a845b0e"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.447769 9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:48.472034 9100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:48.474766 9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:48.474924 9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:48.475208 9100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:48.475579 9100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:48.475630 9100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.475662 9100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:48.475682 9100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.482717 9100 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42311
I20260924 23:06:48.482790 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42311 every 8 connection(s)
I20260924 23:06:48.483165 9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.492565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9100
I20260924 23:06:48.492884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44061
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40211
--webserver_port=46397
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:06:48.493029 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.484022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:54561)
0924 23:06:48.484209 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.484260 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.484752 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.485288 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.485293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.486079 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.486345 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.486352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.487534 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.487535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.488776 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.488778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.489372 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.489374 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.489511 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.490175 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.490187 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.492794 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
W20260924 23:06:48.493234 9248 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40211 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40211: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 23:06:48.493933 9251 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.484099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54561 (local address 127.0.98.62:40211)
0924 23:06:48.484912 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.484918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.484947 (+ 29us) server_negotiation.cc:407] Connection header received
0924 23:06:48.485048 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.485052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.485089 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.485184 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.486480 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.487418 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.488880 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.489277 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.489369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.493706 (+ 4337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54561: BlockingRecv error: recv got EOF from 127.0.98.1:54561 (error 108)
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":114,"threads_started":1}
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.499876 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.493446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:38939)
0924 23:06:48.493567 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.493578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.493620 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.493872 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.493903 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.494434 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.494493 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.494495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.495447 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.495448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.496630 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.496633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.497160 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.497161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.497238 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.497700 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.497716 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.499810 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20260924 23:06:48.500025 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.493567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38939 (local address 127.0.98.62:40211)
0924 23:06:48.493710 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.493712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.493719 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:48.493740 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.493741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.493751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.493811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.494589 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.495346 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.496735 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.497056 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.497143 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.499940 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38939: BlockingRecv error: recv got EOF from 127.0.98.1:38939 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.504961 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.500459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:46837)
0924 23:06:48.500506 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.500516 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.500551 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.500679 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.500680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.500832 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.500879 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.500881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.501463 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.501464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.502400 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.502403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.502754 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.502755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.502816 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.503185 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.503191 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.504918 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 23:06:48.505105 9248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:06:48.505125 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.500522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46837 (local address 127.0.98.62:40211)
0924 23:06:48.500561 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.500564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.500572 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:48.500595 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.500595 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.500605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.500664 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.500953 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.501387 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.502510 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.502680 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.502732 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.505015 (+ 2283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46837: BlockingRecv error: recv got EOF from 127.0.98.1:46837 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:06:48.578742 9253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:48.578948 9253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:48.578970 9253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:48.580042 9253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:48.580075 9253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:48.580101 9253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:06:48.581429 9253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46397
--tserver_master_addrs=127.0.98.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:48.581600 9253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:48.581758 9253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:48.586658 9259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.586678 9258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.586853 9253 server_base.cc:1034] running on GCE node
W20260924 23:06:48.586678 9261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.587100 9253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:48.587361 9253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:48.588560 9253 hybrid_clock.cc:648] HybridClock initialized: now 1790291208588532 us; error 39 us; skew 500 ppm
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291208, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:48.591260 9253 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 23:06:48.591710 9253 webserver.cc:460] Webserver started at http://127.0.98.2:46397/ using document root <none> and password file <none>
I20260924 23:06:48.591958 9253 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:48.592051 9253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:48.593556 9253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:06:48.594252 9268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.594426 9253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:48.594532 9253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "eec3513666f541ff8983395934f86ef0"
format_stamp: "Formatted at 2026-09-24 23:06:47 on dist-test-slave-ph9j"
server_key: "286e99ce690fd1df2987e07bcefbc690"
server_key_iv: "3bb4c126d78d41fa651ad14f82b88774"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.594933 9253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:48.614023 9253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:48.616353 9253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:48.616518 9253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:48.616758 9253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:48.617075 9253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:48.617136 9253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.617177 9253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:48.617205 9253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.623037 9253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44061
I20260924 23:06:48.623086 9381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44061 every 8 connection(s)
I20260924 23:06:48.623409 9253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:06:48.628221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9253
I20260924 23:06:48.628566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38357
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40211
--webserver_port=33465
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:06:48.631271 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.624102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41843 (local address 127.0.98.62:40211)
0924 23:06:48.624168 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.624170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.624381 (+ 211us) server_negotiation.cc:407] Connection header received
0924 23:06:48.624918 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.624919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.624934 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.625000 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.625977 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.626473 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.627463 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.627665 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.627748 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.631221 (+ 3473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41843: BlockingRecv error: recv got EOF from 127.0.98.2:41843 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:06:48.631361 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.624093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:41843)
0924 23:06:48.624268 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.624301 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.624812 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.625073 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.625077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.625642 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.625825 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.625829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.626565 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.626566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.627359 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.627361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.627757 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.627758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.627881 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.628433 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.628444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.631127 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
W20260924 23:06:48.631623 9382 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40211 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40211: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.636869 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.631888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:43711)
0924 23:06:48.631945 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.631958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.632010 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.632125 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.632126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.632285 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.632350 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.632351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.632977 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.632978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.633919 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.633922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.634267 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.634268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.634344 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.634736 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.634742 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.636772 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 23:06:48.636960 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.631958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43711 (local address 127.0.98.62:40211)
0924 23:06:48.632018 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.632020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.632030 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:48.632058 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.632059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.632073 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.632120 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.632428 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.632893 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.634014 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.634194 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.634247 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.636880 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43711: BlockingRecv error: recv got EOF from 127.0.98.2:43711 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.642072 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.637293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:46665)
0924 23:06:48.637345 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.637355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.637392 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.637595 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.637596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.637755 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.637793 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.637795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.638494 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.638494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.639389 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.639392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.639747 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.639748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.639813 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.640179 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.640183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.642000 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 23:06:48.642155 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.637417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46665 (local address 127.0.98.62:40211)
0924 23:06:48.637463 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.637465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.637471 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:06:48.637495 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.637496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.637504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.637569 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.637941 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.638413 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.639493 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.639669 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.639727 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.642081 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46665: BlockingRecv error: recv got EOF from 127.0.98.2:46665 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:06:48.642306 9382 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:06:48.711561 9385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:06:48.711784 9385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:06:48.711822 9385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:06:48.712997 9385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:06:48.713064 9385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:06:48.713109 9385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:06:48.714584 9385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33465
--tserver_master_addrs=127.0.98.62:40211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:06:48.714843 9385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:06:48.715066 9385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:06:48.719911 9393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:06:48.719923 9391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.720137 9385 server_base.cc:1034] running on GCE node
W20260924 23:06:48.720278 9390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:06:48.720482 9385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:06:48.720758 9385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:06:48.721936 9385 hybrid_clock.cc:648] HybridClock initialized: now 1790291208721918 us; error 29 us; skew 500 ppm
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291208, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:06:48.724395 9385 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 23:06:48.724866 9385 webserver.cc:460] Webserver started at http://127.0.98.3:33465/ using document root <none> and password file <none>
I20260924 23:06:48.725106 9385 fs_manager.cc:359] Metadata directory not provided
I20260924 23:06:48.725186 9385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:06:48.726560 9385 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:06:48.727120 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.727301 9385 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:06:48.727382 9385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "9404e0326c164e0683a8fb93a6d323ba"
format_stamp: "Formatted at 2026-09-24 23:06:48 on dist-test-slave-ph9j"
server_key: "9f2f18e437121d808f603903ef19135d"
server_key_iv: "03e79a7e6d40d8f33c8dd8a43f674609"
server_key_version: "encryptionkey@0"
I20260924 23:06:48.727753 9385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:06:48.759534 9385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:06:48.761554 9385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:06:48.761727 9385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:06:48.762005 9385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:06:48.762315 9385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:06:48.762367 9385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.762415 9385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:06:48.762449 9385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:06:48.768262 9385 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38357
I20260924 23:06:48.768375 9513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38357 every 8 connection(s)
I20260924 23:06:48.768687 9385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.774304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9385
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 23:06:48.775869 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.769426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56581 (local address 127.0.98.62:40211)
0924 23:06:48.769488 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.769491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.769727 (+ 236us) server_negotiation.cc:407] Connection header received
0924 23:06:48.770203 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.770204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.770213 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.770265 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.771139 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.771637 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.772601 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.772778 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.772842 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.775824 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56581: BlockingRecv error: recv got EOF from 127.0.98.3:56581 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:06:48.775926 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.769430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:56581)
0924 23:06:48.769608 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.769643 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.770123 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.770316 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.770319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.770865 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.771056 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.771061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.771716 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.771717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.772524 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.772525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.772860 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.772861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.772955 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.773369 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.773376 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.775737 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
W20260924 23:06:48.776175 9514 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40211 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40211: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.780755 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.776356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:52165)
0924 23:06:48.776413 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.776423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.776455 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.776565 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.776566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.776704 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.776752 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.776754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.777328 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.777329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.777861 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.777863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.778312 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.778313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.778362 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.778746 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.778749 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.780661 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 23:06:48.780830 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.776413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52165 (local address 127.0.98.62:40211)
0924 23:06:48.776447 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.776449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.776457 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:48.776509 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.776509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.776516 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.776561 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.776819 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.777244 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.777953 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.778235 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.778306 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.780757 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52165: BlockingRecv error: recv got EOF from 127.0.98.3:52165 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 23:06:48.782352 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.774846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.0.98.62:40211)
0924 23:06:48.774902 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.774904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.775023 (+ 119us) server_negotiation.cc:407] Connection header received
0924 23:06:48.775101 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.775102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.775115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.775168 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.775728 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.776414 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.777318 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.777514 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.777599 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.778867 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:06:48.778872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:06:48.778872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:06:48.778883 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:06:48.780684 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:48.781108 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:48.781112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:48.781113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:48.781276 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:06:48.781545 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:06:48.781546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:06:48.781546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:06:48.781866 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:06:48.782049 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 23:06:48.782127 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 23:06:48.782236 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:48 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:48.785440 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:48.781025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:41179)
0924 23:06:48.781067 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:48.781077 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:48.781108 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:48.781258 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:48.781258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:48.781402 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:48.781443 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.781445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.782354 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.782354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.783054 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:48.783056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.783405 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.783406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:48.783472 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.783805 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:48.783809 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:48.785393 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 23:06:48.785517 9252 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:48.781077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41179 (local address 127.0.98.62:40211)
0924 23:06:48.781109 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:48.781111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:48.781119 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:48.781161 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:48.781162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:48.781171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:48.781234 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:48.781545 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.782265 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.783142 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:48.783334 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:48.783409 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:48.785476 (+ 2067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41179: BlockingRecv error: recv got EOF from 127.0.98.3:41179 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:06:48.785604 9514 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 23:06:49 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:49 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:49.509858 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:49.505536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:48461)
0924 23:06:49.505649 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:49.505664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:49.505712 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:49.505835 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:49.505836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:49.506015 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:49.506062 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:49.506064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.506681 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.506681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:49.507242 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:49.507243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.507571 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.507571 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:49.507608 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:49.508029 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:49.508033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:49.509756 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
W20260924 23:06:49.509995 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:49.505583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48461 (local address 127.0.98.62:40211)
0924 23:06:49.505719 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:49.505721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:49.505730 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:49.505767 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:49.505768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:49.505778 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:49.505823 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:49.506130 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.506615 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.507314 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.507504 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.507559 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:49.509857 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48461: BlockingRecv error: recv got EOF from 127.0.98.1:48461 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
Sep 24 23:06:49 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:49 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:49.647608 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:49.642719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:38485)
0924 23:06:49.642833 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:49.642844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:49.642929 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:49.643062 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:49.643062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:49.643220 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:49.643269 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:49.643270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.643969 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.643971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:49.644543 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:49.644545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.644956 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.644958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:49.645038 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:49.645506 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:49.645510 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:49.647539 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
W20260924 23:06:49.647784 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:49.642787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38485 (local address 127.0.98.62:40211)
0924 23:06:49.642835 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:49.642837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:49.642920 (+ 83us) server_negotiation.cc:407] Connection header received
0924 23:06:49.642976 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:49.642977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:49.642995 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:49.643044 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:49.643334 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.643855 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.644621 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.644809 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.644909 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:49.647706 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38485: BlockingRecv error: recv got EOF from 127.0.98.2:38485 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 23:06:49 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:49 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:49.790824 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:49.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:50205)
0924 23:06:49.786231 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:49.786243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:49.786313 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:49.786462 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:49.786463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:49.786651 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:49.786695 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:49.786697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.787381 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.787383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:49.787945 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:49.787947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.788340 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.788341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:49.788405 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:49.788880 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:49.788885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:49.790717 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":81,"threads_started":1}
W20260924 23:06:49.790866 9519 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:49.786143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50205 (local address 127.0.98.62:40211)
0924 23:06:49.786220 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:49.786223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:49.786297 (+ 74us) server_negotiation.cc:407] Connection header received
0924 23:06:49.786366 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:49.786367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:49.786380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:49.786455 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:49.786782 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.787288 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.788036 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:49.788236 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:49.788326 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:49.790796 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50205: BlockingRecv error: recv got EOF from 127.0.98.3:50205 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 23:06:50 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:50 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:50.514968 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:50.510344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:39753)
0924 23:06:50.510464 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:50.510473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:50.510522 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:50.510660 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:50.510661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:50.510810 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:50.510858 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:50.510859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.511630 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.511631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:50.512154 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:50.512156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.512555 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.512556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:50.512595 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:50.513080 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:50.513084 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:50.514910 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
W20260924 23:06:50.515074 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:50.510395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39753 (local address 127.0.98.62:40211)
0924 23:06:50.510541 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:50.510543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:50.510557 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:06:50.510579 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:50.510579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:50.510589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:50.510655 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:50.510960 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.511537 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.512281 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.512483 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.512543 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:50.514997 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39753: BlockingRecv error: recv got EOF from 127.0.98.1:39753 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":99,"threads_started":1}
Sep 24 23:06:50 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:50 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:50.652410 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:50.648034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:48909)
0924 23:06:50.648165 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:50.648182 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:50.648262 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:50.648389 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:50.648390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:50.648550 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:50.648598 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:50.648599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.649209 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.649210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:50.649731 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:50.649733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.650167 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.650168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:50.650214 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:50.650638 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:50.650643 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:50.652336 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
W20260924 23:06:50.652536 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:50.648130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48909 (local address 127.0.98.62:40211)
0924 23:06:50.648206 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:50.648208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:50.648264 (+ 56us) server_negotiation.cc:407] Connection header received
0924 23:06:50.648307 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:50.648308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:50.648319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:50.648367 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:50.648655 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.649145 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.649798 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.650102 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.650166 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:50.652453 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48909: BlockingRecv error: recv got EOF from 127.0.98.2:48909 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 23:06:50 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:50 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:50.795763 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:50.791355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:60197)
0924 23:06:50.791489 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:50.791521 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:50.791574 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:50.791720 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:50.791721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:50.791880 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:50.791925 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:50.791927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.792557 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.792557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:50.793101 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:50.793102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.793407 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.793408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:50.793445 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:50.793951 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:50.793956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:50.795666 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
W20260924 23:06:50.795801 9523 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:50.791532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60197 (local address 127.0.98.62:40211)
0924 23:06:50.791600 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:50.791602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:50.791610 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:50.791632 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:50.791633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:50.791643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:50.791691 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:50.791991 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.792465 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.793171 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:50.793347 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:50.793397 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:50.795741 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60197: BlockingRecv error: recv got EOF from 127.0.98.3:60197 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 23:06:51 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:51 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:51.519821 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:51.515442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:36265)
0924 23:06:51.515573 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:51.515585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:51.515645 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:51.515784 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:51.515785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:51.515945 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:51.515997 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:51.515999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.516638 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.516638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:51.517195 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:51.517197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.517521 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.517522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:51.517558 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:51.517998 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:51.518003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:51.519717 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
W20260924 23:06:51.519866 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:51.515487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36265 (local address 127.0.98.62:40211)
0924 23:06:51.515632 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:51.515634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:51.515642 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:06:51.515700 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:51.515701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:51.515711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:51.515770 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:51.516059 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.516562 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.517272 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.517457 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.517525 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:51.519793 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36265: BlockingRecv error: recv got EOF from 127.0.98.1:36265 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
Sep 24 23:06:51 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:51 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:51.657483 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:51.652883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:59551)
0924 23:06:51.653036 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:51.653046 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:51.653115 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:51.653246 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:51.653247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:51.653422 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:51.653470 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:51.653471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.654266 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.654268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:51.654788 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:51.654789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.655184 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.655185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:51.655246 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:51.655673 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:51.655677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:51.657417 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
W20260924 23:06:51.657608 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:51.652946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59551 (local address 127.0.98.62:40211)
0924 23:06:51.653041 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:51.653043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:51.653126 (+ 83us) server_negotiation.cc:407] Connection header received
0924 23:06:51.653168 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:51.653169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:51.653181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:51.653222 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:51.653562 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.654184 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.654884 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.655080 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.655171 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:51.657504 (+ 2333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59551: BlockingRecv error: recv got EOF from 127.0.98.2:59551 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 23:06:51 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:51 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:51.801079 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:51.796219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:53753)
0924 23:06:51.796367 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:51.796378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:51.796451 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:51.796593 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:51.796593 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:51.796755 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:51.796804 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:51.796805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.797468 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.797470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:51.798161 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:51.798163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.798529 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.798530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:51.798569 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:51.799231 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:51.799235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:51.801012 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
W20260924 23:06:51.801177 9527 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:51.796325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53753 (local address 127.0.98.62:40211)
0924 23:06:51.796373 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:51.796375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:51.796464 (+ 89us) server_negotiation.cc:407] Connection header received
0924 23:06:51.796502 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:51.796503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:51.796515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:51.796562 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:51.796871 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.797363 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.798260 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:51.798451 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:51.798517 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:51.801107 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53753: BlockingRecv error: recv got EOF from 127.0.98.3:53753 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 23:06:52 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:52 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:52.527047 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:52.520325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:33541)
0924 23:06:52.520461 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:52.520475 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:52.520590 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:52.520797 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:52.520798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:52.521047 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:52.521105 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:52.521106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.522217 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.522218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:52.523250 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:52.523252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.523798 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.523799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:52.523875 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:52.524532 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:52.524537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:52.526950 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
W20260924 23:06:52.527132 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:52.520366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33541 (local address 127.0.98.62:40211)
0924 23:06:52.520520 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:52.520524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:52.520630 (+ 106us) server_negotiation.cc:407] Connection header received
0924 23:06:52.520658 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:52.520658 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:52.520671 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:52.520763 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:52.521200 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.522105 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.523351 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.523710 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.523783 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:52.527050 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33541: BlockingRecv error: recv got EOF from 127.0.98.1:33541 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
Sep 24 23:06:52 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:52 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:52.663086 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:52.658153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:39293)
0924 23:06:52.658328 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:52.658341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:52.658414 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:52.658584 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:52.658585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:52.658765 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:52.658811 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:52.658813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.659477 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.659478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:52.660003 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:52.660004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.660414 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.660415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:52.660472 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:52.660986 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:52.660991 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:52.662977 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
W20260924 23:06:52.663118 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:52.658247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39293 (local address 127.0.98.62:40211)
0924 23:06:52.658330 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:52.658333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:52.658391 (+ 58us) server_negotiation.cc:407] Connection header received
0924 23:06:52.658471 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:52.658472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:52.658485 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:52.658579 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:52.658904 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.659396 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.660113 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.660312 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.660416 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:52.663061 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39293: BlockingRecv error: recv got EOF from 127.0.98.2:39293 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 23:06:52 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:52 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:52.806845 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:52.801675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:57379)
0924 23:06:52.801866 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:52.801905 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:52.802002 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:52.802159 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:52.802160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:52.802345 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:52.802392 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:52.802394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.803155 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.803157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:52.803689 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:52.803691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.804112 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.804113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:52.804171 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:52.804640 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:52.804644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:52.806745 (+ 2101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
W20260924 23:06:52.806897 9531 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:52.801777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57379 (local address 127.0.98.62:40211)
0924 23:06:52.801855 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:52.801857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:52.801983 (+ 126us) server_negotiation.cc:407] Connection header received
0924 23:06:52.802064 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:52.802065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:52.802086 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:52.802153 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:52.802490 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.803066 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.803796 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:52.803983 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:52.804112 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:52.806833 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57379: BlockingRecv error: recv got EOF from 127.0.98.3:57379 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 23:06:53 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:53 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:53.532577 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:53.527756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:54111)
0924 23:06:53.527888 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:53.527900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:53.527957 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:53.528096 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:53.528096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:53.528262 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:53.528317 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:53.528318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.528991 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.528993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:53.529541 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:53.529543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.529994 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.529996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:53.530049 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:53.530577 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:53.530581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:53.532511 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260924 23:06:53.532756 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:53.527807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54111 (local address 127.0.98.62:40211)
0924 23:06:53.527939 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:53.527941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:53.527953 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:53.528003 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:53.528003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:53.528014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:53.528084 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:53.528383 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.528893 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.529614 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.529871 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.529988 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:53.532666 (+ 2678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54111: BlockingRecv error: recv got EOF from 127.0.98.1:54111 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
Sep 24 23:06:53 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:53 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:53.668257 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:53.663594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:53609)
0924 23:06:53.663744 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:53.663757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:53.663826 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:53.664013 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:53.664014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:53.664176 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:53.664224 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:53.664226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.664877 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.664878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:53.665427 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:53.665429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.665756 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.665757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:53.665794 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:53.666250 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:53.666254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:53.668192 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":56,"threads_started":1}
W20260924 23:06:53.668322 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:53.663813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53609 (local address 127.0.98.62:40211)
0924 23:06:53.663880 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:53.663882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:53.663892 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:53.663924 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:53.663925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:53.663936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:53.663984 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:53.664291 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.664798 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.665509 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.665692 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.665743 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:53.668278 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53609: BlockingRecv error: recv got EOF from 127.0.98.2:53609 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 23:06:53 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:53 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:53.816808 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:53.807459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:60519)
0924 23:06:53.807605 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:53.807618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:53.807732 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:53.807909 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:53.807910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:53.808155 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:53.808217 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:53.808219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.808975 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.808976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:53.809589 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:53.809591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.810033 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.810034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:53.810077 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:53.814949 (+ 4872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:53.814955 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:53.816732 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260924 23:06:53.816875 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:53.807600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60519 (local address 127.0.98.62:40211)
0924 23:06:53.807668 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:53.807670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:53.807709 (+ 39us) server_negotiation.cc:407] Connection header received
0924 23:06:53.807812 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:53.807813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:53.807829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:53.807886 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:53.808301 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.808898 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.809662 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:53.809960 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:53.810015 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:53.816829 (+ 6814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60519: BlockingRecv error: recv got EOF from 127.0.98.3:60519 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 23:06:54 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:54 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:54.537756 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:54.533087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:59711)
0924 23:06:54.533217 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:54.533228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:54.533297 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:54.533423 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:54.533424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:54.533584 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:54.533631 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:54.533633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.534574 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.534575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:54.535123 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:54.535125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.535448 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.535449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:54.535487 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:54.535925 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:54.535930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:54.537685 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
W20260924 23:06:54.537878 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:54.533166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59711 (local address 127.0.98.62:40211)
0924 23:06:54.533311 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:54.533313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:54.533323 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:06:54.533349 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:54.533350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:54.533360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:54.533418 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:54.533693 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.534480 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.535195 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.535383 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.535442 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:54.537772 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59711: BlockingRecv error: recv got EOF from 127.0.98.1:59711 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
Sep 24 23:06:54 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:54 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:54.673331 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:54.668836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:53873)
0924 23:06:54.668982 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:54.668993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:54.669074 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:54.669263 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:54.669264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:54.669478 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:54.669526 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:54.669528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.670200 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.670200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:54.670723 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:54.670725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.671035 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.671036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:54.671079 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:54.671531 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:54.671535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:54.673265 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260924 23:06:54.673470 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:54.668901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53873 (local address 127.0.98.62:40211)
0924 23:06:54.668981 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:54.668983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:54.669075 (+ 92us) server_negotiation.cc:407] Connection header received
0924 23:06:54.669137 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:54.669138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:54.669148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:54.669201 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:54.669621 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.670134 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.670794 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.670976 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.671024 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:54.673389 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53873: BlockingRecv error: recv got EOF from 127.0.98.2:53873 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 23:06:54 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:54 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:54.822660 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:54.817372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:42643)
0924 23:06:54.817510 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:54.817521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:54.817570 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:54.817788 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:54.817789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:54.817965 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:54.818043 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:54.818045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.818670 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.818671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:54.819389 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:54.819392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.819757 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.819757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:54.819801 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:54.820253 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:54.820257 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:54.822603 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
W20260924 23:06:54.822780 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:54.817455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42643 (local address 127.0.98.62:40211)
0924 23:06:54.817542 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:54.817543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:54.817555 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:06:54.817697 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:54.817698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:54.817711 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:54.817768 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:54.818116 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.818602 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.819489 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:54.819674 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:54.819732 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:54.822700 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42643: BlockingRecv error: recv got EOF from 127.0.98.3:42643 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 23:06:55 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:55 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:55.542766 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:55.538242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:37453)
0924 23:06:55.538387 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:55.538398 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:55.538452 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:55.538583 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:55.538583 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:55.538739 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:55.538786 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:55.538787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.539439 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.539440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:55.539955 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:55.539957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.540305 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.540306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:55.540354 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:55.540802 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:55.540806 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:55.542670 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
W20260924 23:06:55.542815 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:55.538335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37453 (local address 127.0.98.62:40211)
0924 23:06:55.538442 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:55.538443 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:55.538452 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:55.538498 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:55.538499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:55.538509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:55.538568 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:55.538851 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.539364 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.540027 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.540220 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.540272 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:55.542746 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37453: BlockingRecv error: recv got EOF from 127.0.98.1:37453 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
Sep 24 23:06:55 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:55 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:55.678915 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:55.673817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:58747)
0924 23:06:55.674008 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:55.674021 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:55.674094 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:55.674270 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:55.674271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:55.674454 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:55.674502 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:55.674503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.675207 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.675208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:55.675767 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:55.675769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.676147 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.676148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:55.676205 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:55.676684 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:55.676688 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:55.678855 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
W20260924 23:06:55.679070 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:55.673937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58747 (local address 127.0.98.62:40211)
0924 23:06:55.673999 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:55.674001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:55.674093 (+ 92us) server_negotiation.cc:407] Connection header received
0924 23:06:55.674160 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:55.674161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:55.674175 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:55.674252 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:55.674592 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.675108 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.675854 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.676057 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.676145 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:55.678973 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58747: BlockingRecv error: recv got EOF from 127.0.98.2:58747 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 23:06:55 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:55 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:55.827800 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:55.823185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:43955)
0924 23:06:55.823340 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:55.823357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:55.823415 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:55.823571 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:55.823571 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:55.823734 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:55.823782 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:55.823783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.824392 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.824393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:55.824938 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:55.824940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.825295 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.825296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:55.825339 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:55.825977 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:55.825981 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:55.827743 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
W20260924 23:06:55.827960 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:55.823272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43955 (local address 127.0.98.62:40211)
0924 23:06:55.823317 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:55.823319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:55.823414 (+ 95us) server_negotiation.cc:407] Connection header received
0924 23:06:55.823478 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:55.823479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:55.823492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:55.823547 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:55.823845 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.824324 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.825036 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:55.825222 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:55.825283 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:55.827848 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43955: BlockingRecv error: recv got EOF from 127.0.98.3:43955 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 23:06:56 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:56 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:56.548098 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:56.543244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:35025)
0924 23:06:56.543379 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:56.543389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:56.543460 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:56.543651 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:56.543652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:56.543816 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:56.543863 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:56.543864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.544520 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.544520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:56.545080 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:56.545081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.545429 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.545431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:56.545478 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:56.546052 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:56.546057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:56.547986 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
W20260924 23:06:56.548130 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:56.543362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35025 (local address 127.0.98.62:40211)
0924 23:06:56.543498 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:56.543500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:56.543507 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:06:56.543534 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:56.543534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:56.543543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:56.543608 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:56.543944 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.544452 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.545152 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.545341 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.545398 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:56.548065 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35025: BlockingRecv error: recv got EOF from 127.0.98.1:35025 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
Sep 24 23:06:56 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:56 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:56.684176 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:56.679369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:43547)
0924 23:06:56.679512 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:56.679524 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:56.679591 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:56.679731 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:56.679732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:56.679894 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:56.679943 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:56.679944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.680856 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.680857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:56.681388 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:56.681390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.681753 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.681754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:56.681792 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:56.682334 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:56.682339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:56.684118 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
W20260924 23:06:56.684274 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:56.679425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43547 (local address 127.0.98.62:40211)
0924 23:06:56.679484 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:56.679486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:56.679593 (+ 107us) server_negotiation.cc:407] Connection header received
0924 23:06:56.679651 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:56.679651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:56.679662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:56.679709 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:56.680064 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.680726 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.681486 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.681683 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.681740 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:56.684208 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43547: BlockingRecv error: recv got EOF from 127.0.98.2:43547 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 23:06:56 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:56 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:56.833213 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:56.828361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:49857)
0924 23:06:56.828533 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:56.828544 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:56.828599 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:56.828730 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:56.828731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:56.828888 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:56.828935 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:56.828936 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.829576 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.829577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:56.830178 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:56.830180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.830541 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.830541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:56.830619 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:56.831216 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:56.831220 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:56.833157 (+ 1937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":105,"threads_started":1}
W20260924 23:06:56.833330 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:56.828459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49857 (local address 127.0.98.62:40211)
0924 23:06:56.828522 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:56.828524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:56.828602 (+ 78us) server_negotiation.cc:407] Connection header received
0924 23:06:56.828642 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:56.828643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:56.828656 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:56.828712 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:56.828998 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.829505 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.830275 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:56.830464 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:56.830532 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:56.833250 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49857: BlockingRecv error: recv got EOF from 127.0.98.3:49857 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 23:06:57 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:57 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:57.553617 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:57.548581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:35591)
0924 23:06:57.548719 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:57.548732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:57.548810 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:57.548988 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:57.548990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:57.549153 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:57.549200 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:57.549201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.549864 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.549865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:57.550509 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:57.550511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.550866 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.550867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:57.550908 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:57.551467 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:57.551471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:57.553504 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":50,"threads_started":1}
W20260924 23:06:57.553661 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:57.548678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35591 (local address 127.0.98.62:40211)
0924 23:06:57.548832 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:57.548834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:57.548843 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:57.548864 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:57.548865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:57.548875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:57.548975 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:57.549281 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.549796 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.550601 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.550798 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.550857 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:57.553592 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35591: BlockingRecv error: recv got EOF from 127.0.98.1:35591 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
Sep 24 23:06:57 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:57 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:57.689699 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:57.684744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:56717)
0924 23:06:57.684944 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:57.684956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:57.685058 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:57.685242 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:57.685243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:57.685428 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:57.685474 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:57.685476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.686229 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.686230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:57.686772 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:57.686774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.687152 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.687153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:57.687210 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:57.687722 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:57.687727 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:57.689604 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
W20260924 23:06:57.689776 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:57.684864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56717 (local address 127.0.98.62:40211)
0924 23:06:57.684943 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:57.684945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:57.685038 (+ 93us) server_negotiation.cc:407] Connection header received
0924 23:06:57.685131 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:57.685131 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:57.685145 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:57.685224 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:57.685560 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.686145 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.686869 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.687068 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.687152 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:57.689715 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56717: BlockingRecv error: recv got EOF from 127.0.98.2:56717 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 23:06:57 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:57 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:57.838501 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:57.833744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:50181)
0924 23:06:57.833908 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:57.833920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:57.833989 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:57.834114 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:57.834115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:57.834312 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:57.834358 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:57.834360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.835049 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.835050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:57.835598 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:57.835599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.835978 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.835979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:57.836033 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:57.836511 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:57.836515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:57.838404 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
W20260924 23:06:57.838547 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:57.833813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50181 (local address 127.0.98.62:40211)
0924 23:06:57.833867 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:57.833870 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:57.833969 (+ 99us) server_negotiation.cc:407] Connection header received
0924 23:06:57.834042 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:57.834042 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:57.834058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:57.834109 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:57.834442 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.834950 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.835694 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:57.835890 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:57.835976 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:57.838485 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50181: BlockingRecv error: recv got EOF from 127.0.98.3:50181 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 23:06:58 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:58 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:58.558940 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:58.554140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:47217)
0924 23:06:58.554277 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:58.554289 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:58.554353 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:58.554474 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:58.554475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:58.554630 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:58.554678 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:58.554679 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.555320 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.555321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:58.555853 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:58.555854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.556240 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.556242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:58.556291 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:58.556728 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:58.556733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:58.558880 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260924 23:06:58.559139 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:58.554189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47217 (local address 127.0.98.62:40211)
0924 23:06:58.554334 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:58.554336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:58.554345 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:06:58.554401 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:58.554402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:58.554411 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:58.554464 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:58.554738 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.555230 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.555934 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.556141 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.556192 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:58.559051 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47217: BlockingRecv error: recv got EOF from 127.0.98.1:47217 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
Sep 24 23:06:58 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:58 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:58.694926 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:58.690333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:59921)
0924 23:06:58.690512 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:58.690533 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:58.690603 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:58.690743 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:58.690743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:58.690914 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:58.690974 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:58.690976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.691638 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.691639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:58.692239 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:58.692241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.692589 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.692590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:58.692638 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:58.693094 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:58.693099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:58.694864 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":77,"threads_started":1}
W20260924 23:06:58.694999 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:58.690400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59921 (local address 127.0.98.62:40211)
0924 23:06:58.690496 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:58.690498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:58.690603 (+ 105us) server_negotiation.cc:407] Connection header received
0924 23:06:58.690654 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:58.690655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:58.690666 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:58.690714 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:58.691041 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.691562 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.692317 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.692499 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.692553 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:58.694952 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59921: BlockingRecv error: recv got EOF from 127.0.98.2:59921 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 23:06:58 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:58 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:58.847285 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:58.838951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:52979)
0924 23:06:58.839108 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:58.839118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:58.839187 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:58.839318 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:58.839319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:58.839483 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:58.839529 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:58.839530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.840166 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.840166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:58.840690 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:58.840691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.840991 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.840992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:58.841029 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:58.845195 (+ 4166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:58.845200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:58.847221 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":55,"threads_started":1}
W20260924 23:06:58.847393 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:58.839115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52979 (local address 127.0.98.62:40211)
0924 23:06:58.839188 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:58.839190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:58.839203 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:06:58.839242 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:58.839242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:58.839252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:58.839293 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:58.839595 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.840086 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.840759 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:58.840931 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:58.840979 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:58.847321 (+ 6342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52979: BlockingRecv error: recv got EOF from 127.0.98.3:52979 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 23:06:59 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:59 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:59.564224 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:59.559454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:59533)
0924 23:06:59.559648 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:59.559659 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:59.559714 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:59.559862 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:59.559863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:59.560027 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:59.560074 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:59.560075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.560724 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.560725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:59.561240 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:59.561241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.561597 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.561598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:59.561639 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:59.562261 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:59.562267 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:59.564163 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
W20260924 23:06:59.564370 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:59.559578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59533 (local address 127.0.98.62:40211)
0924 23:06:59.559727 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:59.559729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:59.559754 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:06:59.559777 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:59.559777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:59.559787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:59.559838 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:59.560145 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.560657 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.561321 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.561514 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.561597 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:59.564279 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59533: BlockingRecv error: recv got EOF from 127.0.98.1:59533 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
Sep 24 23:06:59 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:59 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:59.700439 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:59.695445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:60067)
0924 23:06:59.695638 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:59.695650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:59.695741 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:59.695901 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:59.695902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:59.696091 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:59.696139 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:59.696141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.696820 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.696821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:59.697343 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:59.697345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.697779 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.697779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:59.697841 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:59.698347 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:59.698352 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:59.700340 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":112,"threads_started":1}
W20260924 23:06:59.700489 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:59.695571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60067 (local address 127.0.98.62:40211)
0924 23:06:59.695642 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:59.695644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:59.695725 (+ 81us) server_negotiation.cc:407] Connection header received
0924 23:06:59.695807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:59.695807 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:59.695825 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:59.695879 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:59.696239 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.696737 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.697430 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.697701 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.697762 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:59.700427 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60067: BlockingRecv error: recv got EOF from 127.0.98.2:60067 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 23:06:59 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:06:59 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:06:59.852361 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:06:59.847799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:50453)
0924 23:06:59.847942 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:06:59.847953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:06:59.848017 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:06:59.848169 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:06:59.848169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:06:59.848326 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:06:59.848375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:59.848377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.849049 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.849050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:59.849597 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:06:59.849599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.849983 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.849984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:06:59.850028 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:59.850497 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:06:59.850502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:06:59.852299 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260924 23:06:59.852444 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:06:59.847877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50453 (local address 127.0.98.62:40211)
0924 23:06:59.847946 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:06:59.847949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:06:59.848013 (+ 64us) server_negotiation.cc:407] Connection header received
0924 23:06:59.848070 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:06:59.848071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:06:59.848084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:06:59.848133 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:06:59.848436 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.848968 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.849671 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:06:59.849850 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:06:59.849970 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:06:59.852396 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50453: BlockingRecv error: recv got EOF from 127.0.98.3:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 23:07:00 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:00 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:00.569766 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:00.564715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:39019)
0924 23:07:00.564900 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:00.564915 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:00.564986 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:00.565147 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:00.565148 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:00.565326 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:00.565375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:00.565377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.566148 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.566149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:00.566698 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:00.566700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.567073 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.567074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:00.567119 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:00.567640 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:00.567645 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:00.569680 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":90,"threads_started":1}
W20260924 23:07:00.569840 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:00.564819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39019 (local address 127.0.98.62:40211)
0924 23:07:00.564975 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:00.564977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:00.564991 (+ 14us) server_negotiation.cc:407] Connection header received
0924 23:07:00.565042 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:00.565043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:00.565053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:00.565113 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:00.565470 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.566071 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.566794 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.566995 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.567057 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:00.569784 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39019: BlockingRecv error: recv got EOF from 127.0.98.1:39019 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":46,"threads_started":1}
Sep 24 23:07:00 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:00 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:00.705543 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:00.700965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:38901)
0924 23:07:00.701149 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:00.701169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:00.701235 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:00.701370 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:00.701370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:00.701531 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:00.701579 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:00.701580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.702247 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.702248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:00.702788 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:00.702790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.703154 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.703155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:00.703205 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:00.703691 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:00.703696 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:00.705487 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":127,"threads_started":1}
W20260924 23:07:00.705700 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:00.701087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38901 (local address 127.0.98.62:40211)
0924 23:07:00.701152 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:00.701154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:00.701255 (+ 101us) server_negotiation.cc:407] Connection header received
0924 23:07:00.701284 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:00.701285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:00.701296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:00.701349 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:00.701648 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.702176 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.702868 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.703064 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.703123 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:00.705626 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38901: BlockingRecv error: recv got EOF from 127.0.98.2:38901 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 23:07:00 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:00 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:00.857375 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:00.853051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:40589)
0924 23:07:00.853182 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:00.853193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:00.853256 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:00.853389 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:00.853390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:00.853558 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:00.853607 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:00.853608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.854238 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.854239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:00.854766 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:00.854767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.855088 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.855089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:00.855132 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:00.855567 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:00.855571 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:00.857280 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260924 23:07:00.857501 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:00.853101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40589 (local address 127.0.98.62:40211)
0924 23:07:00.853147 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:00.853149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:00.853257 (+ 108us) server_negotiation.cc:407] Connection header received
0924 23:07:00.853306 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:00.853307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:00.853320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:00.853368 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:00.853666 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.854172 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.854840 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:00.855019 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:00.855079 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:00.857422 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40589: BlockingRecv error: recv got EOF from 127.0.98.3:40589 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 23:07:01 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:01 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:01.575130 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:01.570301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:51971)
0924 23:07:01.570460 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:01.570472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:01.570527 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:01.570661 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:01.570661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:01.570823 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:01.570889 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:01.570890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.571654 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.571655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:01.572301 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:01.572307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.572746 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.572747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:01.572791 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:01.573325 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:01.573330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:01.575070 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
W20260924 23:07:01.575233 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:01.570397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51971 (local address 127.0.98.62:40211)
0924 23:07:01.570498 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:01.570500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:01.570513 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:07:01.570575 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:01.570576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:01.570586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:01.570642 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:01.570990 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.571572 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.572384 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.572669 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.572732 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:01.575157 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51971: BlockingRecv error: recv got EOF from 127.0.98.1:51971 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
Sep 24 23:07:01 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:01 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:01.711220 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:01.706090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:56093)
0924 23:07:01.706233 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:01.706274 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:01.706339 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:01.706529 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:01.706530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:01.706714 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:01.706757 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:01.706759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.707474 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.707475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:01.708227 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:01.708229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.708582 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.708583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:01.708640 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:01.709154 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:01.709160 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:01.711110 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
W20260924 23:07:01.711277 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:01.706232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56093 (local address 127.0.98.62:40211)
0924 23:07:01.706308 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:01.706310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:01.706321 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:07:01.706409 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:01.706410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:01.706428 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:01.706474 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:01.706841 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.707346 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.708303 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.708500 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.708572 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:01.711208 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56093: BlockingRecv error: recv got EOF from 127.0.98.2:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 23:07:01 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:01 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:01.863248 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:01.858097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:48939)
0924 23:07:01.858280 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:01.858298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:01.858399 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:01.858585 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:01.858586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:01.858764 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:01.858807 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:01.858808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.859509 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.859511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:01.860094 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:01.860096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.860473 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.860474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:01.860542 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:01.861206 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:01.861216 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:01.863180 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
W20260924 23:07:01.863344 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:01.858184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48939 (local address 127.0.98.62:40211)
0924 23:07:01.858279 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:01.858281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:01.858377 (+ 96us) server_negotiation.cc:407] Connection header received
0924 23:07:01.858495 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:01.858496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:01.858515 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:01.858578 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:01.858895 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.859393 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.860187 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:01.860381 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:01.860473 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:01.863277 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48939: BlockingRecv error: recv got EOF from 127.0.98.3:48939 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 23:07:02 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:02 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:02.580611 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:02.575682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:40123)
0924 23:07:02.575931 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:02.575949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:02.576008 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:02.576166 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:02.576167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:02.576331 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:02.576377 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:02.576378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.577102 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.577103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:02.577624 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:02.577626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.578066 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.578066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:02.578108 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:02.578617 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:02.578622 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:02.580546 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":188,"threads_started":1}
W20260924 23:07:02.580763 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:02.575803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40123 (local address 127.0.98.62:40211)
0924 23:07:02.575944 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:02.575946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:02.575995 (+ 49us) server_negotiation.cc:407] Connection header received
0924 23:07:02.576068 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:02.576069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:02.576079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:02.576146 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:02.576493 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.577002 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.577710 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.577942 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.578045 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:02.580654 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40123: BlockingRecv error: recv got EOF from 127.0.98.1:40123 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
Sep 24 23:07:02 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:02 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:02.716657 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:02.711724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:52769)
0924 23:07:02.711855 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:02.711867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:02.711936 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:02.712086 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:02.712087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:02.712240 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:02.712288 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:02.712290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.713105 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.713107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:02.713636 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:02.713638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.713963 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.713964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:02.714004 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:02.714472 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:02.714477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:02.716599 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
W20260924 23:07:02.716811 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:02.711805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52769 (local address 127.0.98.62:40211)
0924 23:07:02.711871 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:02.711873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:02.711937 (+ 64us) server_negotiation.cc:407] Connection header received
0924 23:07:02.712010 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:02.712011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:02.712022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:02.712066 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:02.712428 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.713005 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.713711 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.713913 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.713998 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:02.716732 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52769: BlockingRecv error: recv got EOF from 127.0.98.2:52769 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 23:07:02 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:02 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:02.868430 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:02.863774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.3:59855)
0924 23:07:02.863922 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:02.863934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:02.863999 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:02.864158 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:02.864159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:02.864322 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:02.864371 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:02.864373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.865060 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.865061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:02.865584 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:02.865585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.865993 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.865994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:02.866049 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:02.866545 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:02.866549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:02.868369 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260924 23:07:02.868726 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:02.863879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59855 (local address 127.0.98.62:40211)
0924 23:07:02.863939 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:02.863941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:02.864008 (+ 67us) server_negotiation.cc:407] Connection header received
0924 23:07:02.864037 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:02.864038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:02.864050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:02.864109 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:02.864481 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.864968 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.865681 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:02.865874 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:02.865963 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:02.868638 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59855: BlockingRecv error: recv got EOF from 127.0.98.3:59855 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 23:07:03 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:03 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:03.586088 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:03.581269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.1:53413)
0924 23:07:03.581455 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:03.581466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:03.581567 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:03.581762 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:03.581763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:03.582037 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:03.582106 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:03.582109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.582777 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.582778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:03.583312 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:03.583313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.583666 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.583667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:03.583704 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:03.584147 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:03.584152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:03.586016 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
W20260924 23:07:03.586256 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:03.581390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53413 (local address 127.0.98.62:40211)
0924 23:07:03.581561 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:03.581564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:03.581575 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:07:03.581644 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:03.581645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:03.581657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:03.581732 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:03.582185 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.582710 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.583395 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.583599 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.583653 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:03.586137 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53413: BlockingRecv error: recv got EOF from 127.0.98.1:53413 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
Sep 24 23:07:03 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 23:07:03 dist-test-slave-ph9j krb5kdc[8517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 23:07:03.721822 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:03.717143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40211 (local address 127.0.98.2:36061)
0924 23:07:03.717296 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:03.717308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:03.717393 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:03.717574 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:03.717574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:03.717753 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:03.717797 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:03.717799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.718545 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.718546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:03.719070 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:03.719071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.719431 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.719431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:03.719489 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:03.719981 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:03.719998 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:03.721755 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40211: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260924 23:07:03.721971 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:03.717318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36061 (local address 127.0.98.62:40211)
0924 23:07:03.717393 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:03.717396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:03.717405 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:07:03.717456 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:03.717456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:03.717469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:03.717565 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:03.717906 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.718416 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.719167 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.719358 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.719432 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:03.721849 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36061: BlockingRecv error: recv got EOF from 127.0.98.2:36061 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:07:03.777413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9100
I20260924 23:07:03.782217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9253
I20260924 23:07:03.786468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9385
I20260924 23:07:03.790910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9030
2026-09-24T23:07:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17483 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T23:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:07:03Z Disabled control of system clock
I20260924 23:07:03.818846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42285
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 23:07:03.900884 9580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:03.901093 9580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:03.901113 9580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:03.902341 9580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:07:03.902382 9580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:03.902395 9580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:03.902408 9580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:07:03.902465 9580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:07:03.903843 9580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42285
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:03.904116 9580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:03.904424 9580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:03.909689 9585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:03.909680 9588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:03.909675 9586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:03.909708 9580 server_base.cc:1034] running on GCE node
I20260924 23:07:03.910286 9580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:03.910576 9580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:03.911767 9580 hybrid_clock.cc:648] HybridClock initialized: now 1790291223911744 us; error 37 us; skew 500 ppm
I20260924 23:07:03.919358 9580 webserver.cc:460] Webserver started at http://127.0.98.62:38871/ using document root <none> and password file <none>
I20260924 23:07:03.919642 9580 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:03.919718 9580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:03.919871 9580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:03.920989 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "b692b1b942df404d8c0eb3dbf00dbc89"
format_stamp: "Formatted at 2026-09-24 23:07:03 on dist-test-slave-ph9j"
server_key: "d3cc472890235edd275ca9ad862a0057"
server_key_iv: "56793eeb134119760dc0305053a52e62"
server_key_version: "encryptionkey@0"
I20260924 23:07:03.921355 9580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "b692b1b942df404d8c0eb3dbf00dbc89"
format_stamp: "Formatted at 2026-09-24 23:07:03 on dist-test-slave-ph9j"
server_key: "d3cc472890235edd275ca9ad862a0057"
server_key_iv: "56793eeb134119760dc0305053a52e62"
server_key_version: "encryptionkey@0"
I20260924 23:07:03.922792 9580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:07:03.923576 9594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:03.923748 9580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:03.923856 9580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "b692b1b942df404d8c0eb3dbf00dbc89"
format_stamp: "Formatted at 2026-09-24 23:07:03 on dist-test-slave-ph9j"
server_key: "d3cc472890235edd275ca9ad862a0057"
server_key_iv: "56793eeb134119760dc0305053a52e62"
server_key_version: "encryptionkey@0"
I20260924 23:07:03.923930 9580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:03.940641 9580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:03.940963 9580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:03.941121 9580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:03.945216 9580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42285
I20260924 23:07:03.945253 9646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42285 every 8 connection(s)
I20260924 23:07:03.945595 9580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:07:03.946310 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:03.948771 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89: Bootstrap starting.
I20260924 23:07:03.949290 9647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:03.949517 9647 log.cc:824] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:03.950232 9647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89: No bootstrap required, opened a new log
I20260924 23:07:03.951695 9647 raft_consensus.cc:348] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42285 } }
I20260924 23:07:03.951826 9647 raft_consensus.cc:374] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:03.951848 9647 raft_consensus.cc:729] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b692b1b942df404d8c0eb3dbf00dbc89, State: Initialized, Role: FOLLOWER
I20260924 23:07:03.951980 9647 consensus_queue.cc:260] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42285 } }
I20260924 23:07:03.952066 9647 raft_consensus.cc:388] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:07:03.952090 9647 raft_consensus.cc:482] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:07:03.952116 9647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:03.952579 9647 raft_consensus.cc:504] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42285 } }
I20260924 23:07:03.952657 9647 leader_election.cc:302] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b692b1b942df404d8c0eb3dbf00dbc89; no voters:
I20260924 23:07:03.952803 9647 leader_election.cc:288] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:07:03.952831 9650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:07:03.952950 9650 raft_consensus.cc:686] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [term 1 LEADER]: Becoming Leader. State: Replica: b692b1b942df404d8c0eb3dbf00dbc89, State: Running, Role: LEADER
I20260924 23:07:03.953089 9647 sys_catalog.cc:564] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:07:03.953114 9650 consensus_queue.cc:237] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42285 } }
I20260924 23:07:03.953509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9580
I20260924 23:07:03.953588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:07:03.953662 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42285 } } }
I20260924 23:07:03.953737 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:03.953796 392 external_mini_cluster.cc:1428] Setting key f9e66d02ba0974f70d768387ac002a7d
I20260924 23:07:03.953986 9659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:07:03.954530 9659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:07:03.954895 9651 sys_catalog.cc:455] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b692b1b942df404d8c0eb3dbf00dbc89. Latest consensus state: current_term: 1 leader_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b692b1b942df404d8c0eb3dbf00dbc89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42285 } } }
I20260924 23:07:03.955003 9651 sys_catalog.cc:458] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89 [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:03.956195 9659 catalog_manager.cc:1347] Generated new cluster ID: 834a038c23cf4ae698c144daafc58013
I20260924 23:07:03.956248 9659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:07:03.959715 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:03.954442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50402 (local address 127.0.98.62:42285)
0924 23:07:03.955324 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:03.955333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:03.955364 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:07:03.955479 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:03.955486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:03.955564 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:03.955708 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:03.955914 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.956919 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.957712 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:03.958207 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:03.958350 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:03.959034 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:03.959038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:03.959038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:03.959229 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:03.959261 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:03.959271 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:03.959345 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:03.959418 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:07:03.959495 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":78,"threads_started":1}
I20260924 23:07:03.977190 9659 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:07:03.977748 9659 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:07:03.987923 9659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b692b1b942df404d8c0eb3dbf00dbc89: Generated new TSK 0
I20260924 23:07:03.988137 9659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:07:03.996701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42285
--builtin_ntp_servers=127.0.98.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 23:07:04.077394 9671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.077616 9671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.077637 9671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.078807 9671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.078850 9671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.078874 9671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:07:04.080193 9671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.080425 9671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.080747 9671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:04.085681 9679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.085698 9671 server_base.cc:1034] running on GCE node
W20260924 23:07:04.085682 9677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.085685 9676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.086261 9671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.086508 9671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.087682 9671 hybrid_clock.cc:648] HybridClock initialized: now 1790291224087661 us; error 36 us; skew 500 ppm
I20260924 23:07:04.089030 9671 webserver.cc:460] Webserver started at http://127.0.98.1:36879/ using document root <none> and password file <none>
I20260924 23:07:04.089289 9671 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.089365 9671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.089524 9671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.090704 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "8ada2f1c38c34c8f8717d8e5cd56e38d"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "5629decdac6c58afafbd3cad5ca34fa4"
server_key_iv: "4cc8f2ef6352599c9a81841e585c432a"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.091081 9671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8ada2f1c38c34c8f8717d8e5cd56e38d"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "5629decdac6c58afafbd3cad5ca34fa4"
server_key_iv: "4cc8f2ef6352599c9a81841e585c432a"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.092398 9671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 23:07:04.093112 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.093278 9671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:07:04.093381 9671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "8ada2f1c38c34c8f8717d8e5cd56e38d"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "5629decdac6c58afafbd3cad5ca34fa4"
server_key_iv: "4cc8f2ef6352599c9a81841e585c432a"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.093456 9671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.133704 9671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.134073 9671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.134230 9671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.134483 9671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:04.134827 9671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:04.134882 9671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.134923 9671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:04.134949 9671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.140892 9671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45545
I20260924 23:07:04.140941 9798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45545 every 8 connection(s)
I20260924 23:07:04.141294 9671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:07:04.142110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9671
I20260924 23:07:04.142210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:07:04.142395 392 external_mini_cluster.cc:1428] Setting key 7c03f4e786467285859716877689658e
I20260924 23:07:04.143394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42285
--builtin_ntp_servers=127.0.98.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 23:07:04.147861 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.142128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40485 (local address 127.0.98.62:42285)
0924 23:07:04.142197 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.142200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.142445 (+ 245us) server_negotiation.cc:407] Connection header received
0924 23:07:04.143084 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.143085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.143097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.143162 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.143952 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.144738 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.146219 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.146569 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.146644 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:04.147489 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.147490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.147490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.147496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.147515 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.147521 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.147614 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.147700 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.147739 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:07:04.148193 9801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.142114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42285 (local address 127.0.98.1:40485)
0924 23:07:04.142300 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:04.142342 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:04.142921 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:04.143230 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:04.143234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:04.143243 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:04.143853 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:04.143858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.144863 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.144865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:04.146133 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:04.146136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.146671 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.146672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:04.146790 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:04.147335 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:07:04.147335 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:04.147365 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:07:04.147390 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:04.147621 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:04.147625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:04.147628 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:07:04.147680 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 23:07:04.147763 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 23:07:04.148780 9799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42285
I20260924 23:07:04.148898 9799 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:04.149091 9799 heartbeater.cc:511] Master 127.0.98.62:42285 requested a full tablet report, sending...
I20260924 23:07:04.149734 9611 ts_manager.cc:194] Registered new tserver with Master: 8ada2f1c38c34c8f8717d8e5cd56e38d (127.0.98.1:45545)
I20260924 23:07:04.150799 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40485
W20260924 23:07:04.226645 9802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.226887 9802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.226908 9802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.228117 9802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.228201 9802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.228247 9802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:07:04.229527 9802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.229707 9802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.229969 9802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:04.234885 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.234925 9808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.234931 9810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.235109 9802 server_base.cc:1034] running on GCE node
I20260924 23:07:04.235327 9802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.235580 9802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.236749 9802 hybrid_clock.cc:648] HybridClock initialized: now 1790291224236729 us; error 44 us; skew 500 ppm
I20260924 23:07:04.238178 9802 webserver.cc:460] Webserver started at http://127.0.98.2:32941/ using document root <none> and password file <none>
I20260924 23:07:04.238467 9802 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.238540 9802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.238706 9802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.239831 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "34686f10ebbc40209fd29356f66b4222"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "1811a353eb57a59f59e10e733c76faf4"
server_key_iv: "73586c33a18cbd9db80fcfe856ac3046"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.240207 9802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "34686f10ebbc40209fd29356f66b4222"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "1811a353eb57a59f59e10e733c76faf4"
server_key_iv: "73586c33a18cbd9db80fcfe856ac3046"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.241585 9802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:07:04.242453 9816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.242666 9802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:07:04.242766 9802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "34686f10ebbc40209fd29356f66b4222"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "1811a353eb57a59f59e10e733c76faf4"
server_key_iv: "73586c33a18cbd9db80fcfe856ac3046"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.242856 9802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.264526 9802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.264875 9802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.265043 9802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.265288 9802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:04.265637 9802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:04.265694 9802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.265735 9802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:04.265764 9802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.271720 9802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42933
I20260924 23:07:04.271766 9929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42933 every 8 connection(s)
I20260924 23:07:04.272105 9802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:07:04.277557 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.272819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42877 (local address 127.0.98.62:42285)
0924 23:07:04.272862 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.272864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.273135 (+ 271us) server_negotiation.cc:407] Connection header received
0924 23:07:04.273677 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.273679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.273691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.273739 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.274508 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.275011 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.276012 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.276207 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.276284 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:04.277226 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.277227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.277228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.277234 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.277258 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.277263 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.277327 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.277424 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.277454 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:07:04.277805 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.272800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42285 (local address 127.0.98.2:42877)
0924 23:07:04.272999 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:04.273034 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:04.273569 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:04.273827 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:04.273833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:04.273847 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:04.274412 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:04.274418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.275115 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.275117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:04.275900 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:04.275902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.276289 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.276290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:04.276430 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:04.277041 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:07:04.277041 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:04.277072 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:07:04.277095 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:04.277328 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:04.277334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:04.277338 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:07:04.277427 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 23:07:04.277531 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260924 23:07:04.278357 9930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42285
I20260924 23:07:04.278450 9930 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:04.278596 9930 heartbeater.cc:511] Master 127.0.98.62:42285 requested a full tablet report, sending...
I20260924 23:07:04.279006 9611 ts_manager.cc:194] Registered new tserver with Master: 34686f10ebbc40209fd29356f66b4222 (127.0.98.2:42933)
I20260924 23:07:04.279034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9802
I20260924 23:07:04.279157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:07:04.279357 392 external_mini_cluster.cc:1428] Setting key 323b8979c17d8fb573cb2459165cd0de
I20260924 23:07:04.279466 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42877
I20260924 23:07:04.280328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42285
--builtin_ntp_servers=127.0.98.20:35187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 23:07:04.362320 9933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.362558 9933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.362591 9933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.363703 9933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.363765 9933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.363804 9933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:07:04.365131 9933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.365392 9933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.365700 9933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:07:04.370929 9933 server_base.cc:1034] running on GCE node
W20260924 23:07:04.370893 9941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.370873 9938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.370863 9939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.371223 9933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.371531 9933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.372674 9933 hybrid_clock.cc:648] HybridClock initialized: now 1790291224372661 us; error 26 us; skew 500 ppm
I20260924 23:07:04.374136 9933 webserver.cc:460] Webserver started at http://127.0.98.3:43295/ using document root <none> and password file <none>
I20260924 23:07:04.374389 9933 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.374470 9933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.374644 9933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.375800 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "ca3e2754506044a8bab4955f48adc4d0"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "c59b7bde9a528c5c251facbc7726f452"
server_key_iv: "2d4aab931c8949df8f93e7582f1cee7e"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.376194 9933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ca3e2754506044a8bab4955f48adc4d0"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "c59b7bde9a528c5c251facbc7726f452"
server_key_iv: "2d4aab931c8949df8f93e7582f1cee7e"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.377590 9933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:07:04.378439 9947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.378618 9933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.378726 9933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "ca3e2754506044a8bab4955f48adc4d0"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "c59b7bde9a528c5c251facbc7726f452"
server_key_iv: "2d4aab931c8949df8f93e7582f1cee7e"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.378821 9933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.388375 9933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.388705 9933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.388863 9933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.389096 9933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:04.389423 9933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:04.389479 9933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.389529 9933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:04.389555 9933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.395530 9933 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45667
I20260924 23:07:04.395596 10060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45667 every 8 connection(s)
I20260924 23:07:04.395989 9933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:07:04.401492 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.396863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46363 (local address 127.0.98.62:42285)
0924 23:07:04.396914 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.396916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.397181 (+ 265us) server_negotiation.cc:407] Connection header received
0924 23:07:04.397731 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.397732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.397744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.397783 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.398464 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.398968 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.399982 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.400191 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.400273 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:04.401141 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.401143 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.401144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.401149 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.401173 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.401177 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.401250 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.401365 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.401399 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:07:04.401786 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.396837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42285 (local address 127.0.98.3:46363)
0924 23:07:04.397046 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:04.397086 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:04.397622 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:04.397861 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:04.397865 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:04.397898 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:04.398368 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:04.398373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.399076 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.399078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:04.399884 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:04.399886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:04.400279 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:04.400281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:04.400418 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:04.400960 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:07:04.400961 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:04.400992 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:07:04.401017 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:04.401258 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:04.401264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:04.401268 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:07:04.401369 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 23:07:04.401502 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260924 23:07:04.402515 10061 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42285
I20260924 23:07:04.402616 10061 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:04.402853 10061 heartbeater.cc:511] Master 127.0.98.62:42285 requested a full tablet report, sending...
I20260924 23:07:04.403290 9611 ts_manager.cc:194] Registered new tserver with Master: ca3e2754506044a8bab4955f48adc4d0 (127.0.98.3:45667)
I20260924 23:07:04.403627 9611 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46363
I20260924 23:07:04.405164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 9933
I20260924 23:07:04.405246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:07:04.405464 392 external_mini_cluster.cc:1428] Setting key efb151f4b078a6760f3586965d0cde78
I20260924 23:07:04.406657 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 23:07:04.408739 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:04.408066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.0.98.62:42285)
0924 23:07:04.408132 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.408134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.408350 (+ 216us) server_negotiation.cc:407] Connection header received
0924 23:07:04.408419 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.408420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.408433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.408507 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.408658 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50408: BlockingRecv error: recv got EOF from 127.0.0.1:50408 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:07:04.408794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9671
I20260924 23:07:04.413763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9802
I20260924 23:07:04.418123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9933
I20260924 23:07:04.422385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 9580
2026-09-24T23:07:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (630 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T23:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:07:04Z Disabled control of system clock
I20260924 23:07:04.448555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43705
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:07:04.532030 10072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.532264 10072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.532325 10072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.533557 10072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:07:04.533599 10072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.533612 10072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.533624 10072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:07:04.533638 10072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:07:04.535048 10072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.535254 10072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.535506 10072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:04.540472 10080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.540453 10078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.540445 10077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.540803 10072 server_base.cc:1034] running on GCE node
I20260924 23:07:04.541081 10072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.541375 10072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.542555 10072 hybrid_clock.cc:648] HybridClock initialized: now 1790291224542522 us; error 44 us; skew 500 ppm
I20260924 23:07:04.543906 10072 webserver.cc:460] Webserver started at http://127.0.98.62:44807/ using document root <none> and password file <none>
I20260924 23:07:04.544174 10072 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.544246 10072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.544404 10072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.545512 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "0cb3755d145d43d980ea92ac478a286b"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "a9654b12cc932dcfb391b112aed03482"
server_key_iv: "7d1a5ae0220acc96cc3cee81922e8f03"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.545917 10072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cb3755d145d43d980ea92ac478a286b"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "a9654b12cc932dcfb391b112aed03482"
server_key_iv: "7d1a5ae0220acc96cc3cee81922e8f03"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.547286 10072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:04.548237 10086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.548568 10072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:04.548704 10072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "0cb3755d145d43d980ea92ac478a286b"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "a9654b12cc932dcfb391b112aed03482"
server_key_iv: "7d1a5ae0220acc96cc3cee81922e8f03"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.548794 10072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.552246 10072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.552523 10072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.552662 10072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.556457 10072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43705
I20260924 23:07:04.556509 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43705 every 8 connection(s)
I20260924 23:07:04.556849 10072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:07:04.557564 10139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:04.560518 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b: Bootstrap starting.
I20260924 23:07:04.561131 10139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:04.561484 10139 log.cc:824] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:04.562295 10139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b: No bootstrap required, opened a new log
I20260924 23:07:04.562901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10072
I20260924 23:07:04.562999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:07:04.563185 392 external_mini_cluster.cc:1428] Setting key 834f6138e6b907e599bb9b3884fa1ea8
I20260924 23:07:04.564550 10139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3755d145d43d980ea92ac478a286b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43705 } }
I20260924 23:07:04.564775 10139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:04.564851 10139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb3755d145d43d980ea92ac478a286b, State: Initialized, Role: FOLLOWER
I20260924 23:07:04.565004 10139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3755d145d43d980ea92ac478a286b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43705 } }
I20260924 23:07:04.565131 10139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:07:04.565181 10139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:07:04.565260 10139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:04.566155 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.563785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.0.98.62:43705)
0924 23:07:04.563992 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.563999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.564023 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:07:04.564527 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.564534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.564597 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.564723 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.565679 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.565683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.565684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.565687 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.565731 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.565742 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.565806 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.565913 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.566010 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260924 23:07:04.566167 10139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3755d145d43d980ea92ac478a286b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43705 } }
I20260924 23:07:04.566323 10139 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb3755d145d43d980ea92ac478a286b; no voters:
I20260924 23:07:04.566484 10139 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:07:04.566767 10144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:07:04.566814 10139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:07:04.566884 10144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [term 1 LEADER]: Becoming Leader. State: Replica: 0cb3755d145d43d980ea92ac478a286b, State: Running, Role: LEADER
I20260924 23:07:04.566963 10144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3755d145d43d980ea92ac478a286b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43705 } }
I20260924 23:07:04.567346 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb3755d145d43d980ea92ac478a286b. Latest consensus state: current_term: 1 leader_uuid: "0cb3755d145d43d980ea92ac478a286b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3755d145d43d980ea92ac478a286b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43705 } } }
I20260924 23:07:04.567358 10145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb3755d145d43d980ea92ac478a286b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb3755d145d43d980ea92ac478a286b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43705 } } }
I20260924 23:07:04.567453 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:04.567464 10145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:04.568115 10153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:07:04.568648 10153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:07:04.570197 10153 catalog_manager.cc:1347] Generated new cluster ID: f03ffc0bebb741debb807c7fdc7de3fc
I20260924 23:07:04.570254 10153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:07:04.574120 10153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:07:04.574283 10153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:07:04.579217 10153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb3755d145d43d980ea92ac478a286b: Generated new TSK 0
I20260924 23:07:04.579396 10153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:07:04.588068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43705
--builtin_ntp_servers=127.0.98.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:07:04.669643 10163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.669847 10163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.669868 10163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.671042 10163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.671082 10163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.671108 10163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:07:04.672501 10163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.672724 10163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.673038 10163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:04.678781 10168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.678769 10171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.678769 10169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.679216 10163 server_base.cc:1034] running on GCE node
I20260924 23:07:04.679414 10163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.679659 10163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.680826 10163 hybrid_clock.cc:648] HybridClock initialized: now 1790291224680809 us; error 33 us; skew 500 ppm
I20260924 23:07:04.682211 10163 webserver.cc:460] Webserver started at http://127.0.98.1:35965/ using document root <none> and password file <none>
I20260924 23:07:04.682475 10163 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.682546 10163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.682708 10163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.683755 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "19cae44e3aaa49018b01d38a135a18fe"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "a46205e76d870775f4c605f1f44cb083"
server_key_iv: "ac92e2a11294d5ecda96beabbb064fac"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.684137 10163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "19cae44e3aaa49018b01d38a135a18fe"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "a46205e76d870775f4c605f1f44cb083"
server_key_iv: "ac92e2a11294d5ecda96beabbb064fac"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.685562 10163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:07:04.686457 10177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.686635 10163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:07:04.686740 10163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "19cae44e3aaa49018b01d38a135a18fe"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "a46205e76d870775f4c605f1f44cb083"
server_key_iv: "ac92e2a11294d5ecda96beabbb064fac"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.686817 10163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.690223 10163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.690552 10163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.690714 10163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.690933 10163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:04.691252 10163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:04.691308 10163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.691349 10163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:04.691375 10163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.697211 10163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37813
I20260924 23:07:04.697292 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37813 every 8 connection(s)
I20260924 23:07:04.697615 10163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:07:04.701099 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.698415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35569 (local address 127.0.98.62:43705)
0924 23:07:04.698524 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.698526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.698744 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:07:04.699358 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.699359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.699367 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.699426 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.700806 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.700807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.700808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.700813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.700831 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.700836 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.700896 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.701004 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.701023 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:07:04.701350 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.698333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43705 (local address 127.0.98.1:35569)
0924 23:07:04.698613 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:04.698649 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:04.699234 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:04.699534 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:04.699537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:04.699546 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:04.700635 (+ 1089us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:07:04.700635 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:04.700667 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:07:04.700697 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:04.700891 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:04.700895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:04.700900 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:07:04.701002 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 23:07:04.701071 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260924 23:07:04.701850 10291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43705
I20260924 23:07:04.701997 10291 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:04.702154 10291 heartbeater.cc:511] Master 127.0.98.62:43705 requested a full tablet report, sending...
I20260924 23:07:04.702672 10103 ts_manager.cc:194] Registered new tserver with Master: 19cae44e3aaa49018b01d38a135a18fe (127.0.98.1:37813)
I20260924 23:07:04.702868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10163
I20260924 23:07:04.702975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:07:04.703285 392 external_mini_cluster.cc:1428] Setting key 8e482fcd47ad2d5fdeec2fdbde669aa9
I20260924 23:07:04.704133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43705
--builtin_ntp_servers=127.0.98.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:07:04.784039 10294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.784274 10294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.784297 10294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.785406 10294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.785444 10294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.785468 10294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:07:04.786828 10294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.787024 10294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.787256 10294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:04.792374 10302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.792460 10294 server_base.cc:1034] running on GCE node
W20260924 23:07:04.792382 10299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.792400 10300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.792927 10294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.793200 10294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.794340 10294 hybrid_clock.cc:648] HybridClock initialized: now 1790291224794316 us; error 46 us; skew 500 ppm
I20260924 23:07:04.795554 10294 webserver.cc:460] Webserver started at http://127.0.98.2:41831/ using document root <none> and password file <none>
I20260924 23:07:04.795790 10294 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.795836 10294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.796005 10294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.797055 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "33676cebaf8f4de88382201e9c6102d8"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "af6db4674ceda8ab2ac6832214c02612"
server_key_iv: "51f0141c40a0a0a246f248a4814bf511"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.797408 10294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33676cebaf8f4de88382201e9c6102d8"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "af6db4674ceda8ab2ac6832214c02612"
server_key_iv: "51f0141c40a0a0a246f248a4814bf511"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.798763 10294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:07:04.799458 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.799636 10294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:07:04.799722 10294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "33676cebaf8f4de88382201e9c6102d8"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "af6db4674ceda8ab2ac6832214c02612"
server_key_iv: "51f0141c40a0a0a246f248a4814bf511"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.799795 10294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.803287 10294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.803562 10294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.803709 10294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.803916 10294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:04.804219 10294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:04.804253 10294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.804306 10294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:04.804334 10294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.810380 10294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44091
I20260924 23:07:04.810431 10421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44091 every 8 connection(s)
I20260924 23:07:04.810777 10294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:07:04.814112 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.811602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45767 (local address 127.0.98.62:43705)
0924 23:07:04.811666 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.811667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.811923 (+ 256us) server_negotiation.cc:407] Connection header received
0924 23:07:04.812472 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.812473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.812480 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.812541 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.813820 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.813822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.813822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.813827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.813844 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.813848 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.813924 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.814016 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.814031 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:07:04.814309 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.811596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43705 (local address 127.0.98.2:45767)
0924 23:07:04.811799 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:04.811832 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:04.812357 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:04.812627 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:04.812630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:04.812638 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:04.813660 (+ 1022us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:07:04.813661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:04.813688 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:07:04.813716 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:04.813922 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:04.813927 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:04.813930 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:07:04.814026 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 23:07:04.814088 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260924 23:07:04.814821 10422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43705
I20260924 23:07:04.814929 10422 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:04.815085 10422 heartbeater.cc:511] Master 127.0.98.62:43705 requested a full tablet report, sending...
I20260924 23:07:04.815454 10103 ts_manager.cc:194] Registered new tserver with Master: 33676cebaf8f4de88382201e9c6102d8 (127.0.98.2:44091)
I20260924 23:07:04.818547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10294
I20260924 23:07:04.818634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:07:04.818790 392 external_mini_cluster.cc:1428] Setting key 85479e4d66c7828100eca9083eea0c38
I20260924 23:07:04.819623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43705
--builtin_ntp_servers=127.0.98.20:34863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 23:07:04.900594 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:04.900804 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:04.900826 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:04.902019 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:04.902060 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:04.902086 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:07:04.903522 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:04.903775 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:04.904080 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:04.909154 10433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.909162 10430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:04.909162 10431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:04.909332 10425 server_base.cc:1034] running on GCE node
I20260924 23:07:04.909718 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:04.910024 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:04.911164 10425 hybrid_clock.cc:648] HybridClock initialized: now 1790291224911142 us; error 40 us; skew 500 ppm
I20260924 23:07:04.912358 10425 webserver.cc:460] Webserver started at http://127.0.98.3:36453/ using document root <none> and password file <none>
I20260924 23:07:04.912632 10425 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:04.912704 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:04.912858 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:04.913964 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "237d37a7dab2451aa2a25ad0cdd6219b"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "d58181918d2fbe3a666b0a4b68f56409"
server_key_iv: "11b99f6ea577df59b21d84f7261d88f0"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.914319 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "237d37a7dab2451aa2a25ad0cdd6219b"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "d58181918d2fbe3a666b0a4b68f56409"
server_key_iv: "11b99f6ea577df59b21d84f7261d88f0"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.915553 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:07:04.916232 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.916395 10425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.916497 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "237d37a7dab2451aa2a25ad0cdd6219b"
format_stamp: "Formatted at 2026-09-24 23:07:04 on dist-test-slave-ph9j"
server_key: "d58181918d2fbe3a666b0a4b68f56409"
server_key_iv: "11b99f6ea577df59b21d84f7261d88f0"
server_key_version: "encryptionkey@0"
I20260924 23:07:04.916569 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:04.919893 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:04.920137 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:04.920282 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:04.920475 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:04.920768 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:04.920820 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.920859 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:04.920887 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:04.926676 10425 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38125
I20260924 23:07:04.926707 10552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38125 every 8 connection(s)
I20260924 23:07:04.927050 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:07:04.930220 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.927805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46785 (local address 127.0.98.62:43705)
0924 23:07:04.927850 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.927852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.928095 (+ 243us) server_negotiation.cc:407] Connection header received
0924 23:07:04.928600 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.928601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.928607 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.928666 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.929917 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:04.929918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:04.929919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:07:04.929923 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:04.929940 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:07:04.929945 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:04.930034 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:04.930135 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 23:07:04.930153 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:07:04.930414 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:04.927789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43705 (local address 127.0.98.3:46785)
0924 23:07:04.927977 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:04.928008 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:04.928487 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:04.928736 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:04.928740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:04.928749 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:04.929736 (+ 987us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:07:04.929737 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:04.929764 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:07:04.929791 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:04.930034 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:04.930041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:04.930044 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:07:04.930142 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 23:07:04.930202 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 23:07:04.930922 10553 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43705
I20260924 23:07:04.930999 10553 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:04.931180 10553 heartbeater.cc:511] Master 127.0.98.62:43705 requested a full tablet report, sending...
I20260924 23:07:04.931558 10103 ts_manager.cc:194] Registered new tserver with Master: 237d37a7dab2451aa2a25ad0cdd6219b (127.0.98.3:38125)
I20260924 23:07:04.934083 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10425
I20260924 23:07:04.934155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:07:04.934312 392 external_mini_cluster.cc:1428] Setting key ffababbba70594104c41206142df4e23
I20260924 23:07:04.935092 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 23:07:04.936718 10143 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:07:04.936365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.0.98.62:43705)
0924 23:07:04.936405 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:04.936406 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:04.936491 (+ 85us) server_negotiation.cc:407] Connection header received
0924 23:07:04.936540 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:04.936540 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:04.936544 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:04.936593 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:04.936676 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49752: BlockingRecv error: recv got EOF from 127.0.0.1:49752 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:07:04.936941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10163
I20260924 23:07:04.941192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10294
I20260924 23:07:04.945403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10425
I20260924 23:07:04.949476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10072
2026-09-24T23:07:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (527 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:07:05 dist-test-slave-ph9j krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:07:05 dist-test-slave-ph9j krb5kdc[10563](info): set up 2 sockets
Sep 24 23:07:05 dist-test-slave-ph9j krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260924 23:07:05.994968 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:07:06.120507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36147
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:06.201694 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:06.201972 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:06.202010 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:06.203194 10576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:07:06.203235 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:06.203250 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:06.203263 10576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:07:06.203275 10576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:07:06.204767 10576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:06.205035 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:06.205266 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:06.210173 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:06.210181 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:06.210214 10576 server_base.cc:1034] running on GCE node
W20260924 23:07:06.210201 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:06.210631 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:06.210991 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:06.212193 10576 hybrid_clock.cc:648] HybridClock initialized: now 1790291226212173 us; error 37 us; skew 500 ppm
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:06.214772 10576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:07:06.215261 10576 webserver.cc:460] Webserver started at http://127.0.98.62:44897/ using document root <none> and password file <none>
I20260924 23:07:06.215494 10576 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:06.215538 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:06.215693 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:06.216763 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "7a3206d6e8b44e8886de8d1f454e171e"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "d0f3c6844fe8bf5cbdd8da0987e80e0f"
server_key_iv: "1c5d9bc6dcb75dfbf917b004f3007cad"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.217127 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a3206d6e8b44e8886de8d1f454e171e"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "d0f3c6844fe8bf5cbdd8da0987e80e0f"
server_key_iv: "1c5d9bc6dcb75dfbf917b004f3007cad"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.218514 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:07:06.219316 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.219527 10576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:06.219619 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "7a3206d6e8b44e8886de8d1f454e171e"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "d0f3c6844fe8bf5cbdd8da0987e80e0f"
server_key_iv: "1c5d9bc6dcb75dfbf917b004f3007cad"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.219671 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:06.260461 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:06.262662 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:06.262776 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:06.266579 10576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36147
I20260924 23:07:06.266637 10643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36147 every 8 connection(s)
I20260924 23:07:06.266997 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:07:06.268075 10644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:06.270609 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e: Bootstrap starting.
I20260924 23:07:06.271237 10644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:06.271559 10644 log.cc:824] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:06.272327 10644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e: No bootstrap required, opened a new log
I20260924 23:07:06.274143 10644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3206d6e8b44e8886de8d1f454e171e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260924 23:07:06.274318 10644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:06.274374 10644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3206d6e8b44e8886de8d1f454e171e, State: Initialized, Role: FOLLOWER
I20260924 23:07:06.274580 10644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3206d6e8b44e8886de8d1f454e171e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260924 23:07:06.274694 10644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:07:06.274744 10644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:07:06.274796 10644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:06.275662 10644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3206d6e8b44e8886de8d1f454e171e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260924 23:07:06.275810 10644 leader_election.cc:302] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3206d6e8b44e8886de8d1f454e171e; no voters:
I20260924 23:07:06.275956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10576
I20260924 23:07:06.276007 10644 leader_election.cc:288] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:07:06.276031 10647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:07:06.276108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:07:06.276165 10647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [term 1 LEADER]: Becoming Leader. State: Replica: 7a3206d6e8b44e8886de8d1f454e171e, State: Running, Role: LEADER
I20260924 23:07:06.276309 10644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:07:06.276319 392 external_mini_cluster.cc:1428] Setting key fad9ecae65c2957697f2f023adc22425
I20260924 23:07:06.276292 10647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3206d6e8b44e8886de8d1f454e171e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } }
I20260924 23:07:06.277024 10648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3206d6e8b44e8886de8d1f454e171e. Latest consensus state: current_term: 1 leader_uuid: "7a3206d6e8b44e8886de8d1f454e171e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3206d6e8b44e8886de8d1f454e171e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } } }
I20260924 23:07:06.277280 10648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:06.277204 10649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3206d6e8b44e8886de8d1f454e171e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3206d6e8b44e8886de8d1f454e171e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36147 } } }
I20260924 23:07:06.277540 10660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:07:06.277570 10649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:06.278136 10660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:07:06.279788 10660 catalog_manager.cc:1347] Generated new cluster ID: f2669481e972412996720bc5dd2bc66c
I20260924 23:07:06.279844 10660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:06.287768 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.276833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.0.98.62:36147)
0924 23:07:06.277060 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.277071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.277612 (+ 541us) server_negotiation.cc:407] Connection header received
0924 23:07:06.277751 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.277759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.277836 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.278029 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:06.278919 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.280162 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.280962 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.281386 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.281503 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.283875 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:06.283879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:06.283880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:06.283892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:06.286113 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.286523 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.286528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.286528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.286634 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.286850 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.286851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.286852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.287111 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:06.287290 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.287389 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.287552 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260924 23:07:06.300689 10660 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:07:06.301297 10660 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:07:06.304168 10660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e: Generated new TSK 0
I20260924 23:07:06.304405 10660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:07:06.334589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:06.420640 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:06.420849 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:06.420873 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:06.422088 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:06.422130 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:06.422158 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:07:06.423663 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:06.423887 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:06.424136 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:06.429275 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:06.429311 10670 server_base.cc:1034] running on GCE node
W20260924 23:07:06.429281 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:06.429281 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:06.429661 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:06.429957 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:06.431126 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790291226431111 us; error 30 us; skew 500 ppm
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:06.433709 10670 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:07:06.434175 10670 webserver.cc:460] Webserver started at http://127.0.98.1:41753/ using document root <none> and password file <none>
I20260924 23:07:06.434350 10670 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:06.434391 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:06.434487 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:06.435503 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "f61c1687d28d41ef86fabf62da32f9c3"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "5ad34452497a2837f61c5d343d8b7c4d"
server_key_iv: "aa4864efac3af2f36f99681da8f511ad"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.435792 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f61c1687d28d41ef86fabf62da32f9c3"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "5ad34452497a2837f61c5d343d8b7c4d"
server_key_iv: "aa4864efac3af2f36f99681da8f511ad"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.437098 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:07:06.437848 10685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.438041 10670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:07:06.438145 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "f61c1687d28d41ef86fabf62da32f9c3"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "5ad34452497a2837f61c5d343d8b7c4d"
server_key_iv: "aa4864efac3af2f36f99681da8f511ad"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.438225 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:06.450866 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:06.453289 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:06.453461 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:06.453708 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:06.454128 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:06.454185 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.454219 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:06.454264 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.460099 10670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42875
I20260924 23:07:06.460139 10798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42875 every 8 connection(s)
I20260924 23:07:06.460489 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:06.469923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10670
I20260924 23:07:06.470026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:07:06.470144 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.461268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47117 (local address 127.0.98.62:36147)
0924 23:07:06.461316 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.461318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.461528 (+ 210us) server_negotiation.cc:407] Connection header received
0924 23:07:06.462081 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.462082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.462092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.462154 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:06.463109 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.463626 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.464625 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.464808 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.464859 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.467090 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:06.467092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:06.467092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:06.467100 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:06.468473 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.469005 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.469006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.469006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.469064 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.469315 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.469316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.469316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.469429 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:06.469514 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.469826 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.469980 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:07:06.470235 392 external_mini_cluster.cc:1428] Setting key 70f96e786350021ddc36771e17a15667
I20260924 23:07:06.470232 10801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.461216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.1:47117)
0924 23:07:06.461378 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:06.461420 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:06.461972 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:06.462231 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:06.462234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:06.462820 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:06.463021 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.463026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.463726 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.463727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.464547 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.464549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.464901 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.464902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.465033 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.465521 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:06.465528 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:06.466964 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:06.468601 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:06.468609 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:06.468611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:06.468919 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:06.469132 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:06.469132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:06.469133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:06.469247 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:06.469497 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:06.469503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:06.469648 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 23:07:06.469772 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 23:07:06.469974 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260924 23:07:06.470683 10799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36147
I20260924 23:07:06.470775 10799 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:06.470995 10799 heartbeater.cc:511] Master 127.0.98.62:36147 requested a full tablet report, sending...
I20260924 23:07:06.471599 10607 ts_manager.cc:194] Registered new tserver with Master: f61c1687d28d41ef86fabf62da32f9c3 (127.0.98.1:42875)
I20260924 23:07:06.472499 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47117
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:07:06.497669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:06.583144 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:06.583341 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:06.583364 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:06.584497 10804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:06.584538 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:06.584564 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:07:06.586170 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:06.586396 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:06.586615 10804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:06.591522 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:06.591535 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:06.591694 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:06.591677 10804 server_base.cc:1034] running on GCE node
I20260924 23:07:06.591943 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:06.592203 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:06.593348 10804 hybrid_clock.cc:648] HybridClock initialized: now 1790291226593339 us; error 32 us; skew 500 ppm
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:06.596100 10804 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:07:06.596602 10804 webserver.cc:460] Webserver started at http://127.0.98.2:39037/ using document root <none> and password file <none>
I20260924 23:07:06.596829 10804 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:06.596900 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:06.597054 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:06.598182 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "06f4bbc048e5404f824c6f1077192ff0"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "5ca0bc6244dfdad0041da65938935a41"
server_key_iv: "830f685536fc5269a511ba5f397371c5"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.598562 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "06f4bbc048e5404f824c6f1077192ff0"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "5ca0bc6244dfdad0041da65938935a41"
server_key_iv: "830f685536fc5269a511ba5f397371c5"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.599922 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 23:07:06.600730 10819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.600924 10804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:07:06.601024 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "06f4bbc048e5404f824c6f1077192ff0"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "5ca0bc6244dfdad0041da65938935a41"
server_key_iv: "830f685536fc5269a511ba5f397371c5"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.601120 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:06.621659 10804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:06.624533 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:06.624697 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:06.624963 10804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:06.625329 10804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:06.625363 10804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.625386 10804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:06.625404 10804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.631147 10804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41371
I20260924 23:07:06.631186 10932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41371 every 8 connection(s)
I20260924 23:07:06.631491 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:07:06.632429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10804
I20260924 23:07:06.632553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:07:06.632759 392 external_mini_cluster.cc:1428] Setting key 768a96486ef5f0fa2e378c7312b9706b
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 23:07:06.642628 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.632210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46339 (local address 127.0.98.62:36147)
0924 23:07:06.632291 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.632293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.632442 (+ 149us) server_negotiation.cc:407] Connection header received
0924 23:07:06.633377 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.633378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.633386 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.633452 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:06.634878 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.635430 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.636752 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.637069 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.637146 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.639568 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:06.639570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:06.639571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:06.639581 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:06.641146 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.641603 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.641605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.641605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.641666 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.641961 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.641962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.641963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.642106 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:06.642170 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.642448 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.642536 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 23:07:06.642810 10935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.632137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.2:46339)
0924 23:07:06.632290 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:06.632341 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:06.633054 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:06.633573 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:06.633578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:06.634475 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:06.634767 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.634773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.635527 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.635528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.636659 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.636662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.637157 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.637158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.637262 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.637808 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:06.637817 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:06.639452 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:06.641253 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:06.641259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:06.641261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:06.641508 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:06.641735 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:06.641736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:06.641736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:06.641880 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:06.642161 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:06.642167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:06.642310 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 23:07:06.642415 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 23:07:06.642575 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260924 23:07:06.643287 10933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36147
I20260924 23:07:06.643388 10933 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:06.643620 10933 heartbeater.cc:511] Master 127.0.98.62:36147 requested a full tablet report, sending...
I20260924 23:07:06.644011 10607 ts_manager.cc:194] Registered new tserver with Master: 06f4bbc048e5404f824c6f1077192ff0 (127.0.98.2:41371)
I20260924 23:07:06.644354 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:07:06.661026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:06.748189 10938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:06.748386 10938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:06.748409 10938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:06.749529 10938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:06.749569 10938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:06.749595 10938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:07:06.751050 10938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:06.751286 10938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:06.751506 10938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:06.756587 10944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:06.756641 10946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:06.756594 10943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:06.756806 10938 server_base.cc:1034] running on GCE node
I20260924 23:07:06.757031 10938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:06.757272 10938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:06.758435 10938 hybrid_clock.cc:648] HybridClock initialized: now 1790291226758422 us; error 35 us; skew 500 ppm
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:06.760962 10938 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:07:06.761433 10938 webserver.cc:460] Webserver started at http://127.0.98.3:43687/ using document root <none> and password file <none>
I20260924 23:07:06.761660 10938 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:06.761729 10938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:06.761931 10938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:07:06.762996 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "55a4405ef6ed46c78dcbae099024f0d1"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "b80d0ccad2e57d6541bfdd5305898dda"
server_key_iv: "b5b6eb0dbf060cd48949ece0d90a1f67"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.763367 10938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55a4405ef6ed46c78dcbae099024f0d1"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "b80d0ccad2e57d6541bfdd5305898dda"
server_key_iv: "b5b6eb0dbf060cd48949ece0d90a1f67"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.764707 10938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:07:06.765475 10953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.765650 10938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:07:06.765749 10938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "55a4405ef6ed46c78dcbae099024f0d1"
format_stamp: "Formatted at 2026-09-24 23:07:06 on dist-test-slave-ph9j"
server_key: "b80d0ccad2e57d6541bfdd5305898dda"
server_key_iv: "b5b6eb0dbf060cd48949ece0d90a1f67"
server_key_version: "encryptionkey@0"
I20260924 23:07:06.765837 10938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:06.776387 10938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:06.778895 10938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:06.779083 10938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:06.779327 10938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:06.779656 10938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:06.779711 10938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.779752 10938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:06.779779 10938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:06.785672 10938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45999
I20260924 23:07:06.785710 11066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45999 every 8 connection(s)
I20260924 23:07:06.786108 10938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:07:06 dist-test-slave-ph9j krb5kdc[10563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:06.795897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 10938
I20260924 23:07:06.796010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:07:06.796213 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.786907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46267 (local address 127.0.98.62:36147)
0924 23:07:06.786976 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.786978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.787207 (+ 229us) server_negotiation.cc:407] Connection header received
0924 23:07:06.787921 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.787923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.787946 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.788024 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:06.789122 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.789615 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.790703 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.790898 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.791021 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.793221 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:06.793223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:06.793223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:06.793229 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:06.794621 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.795167 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.795168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.795168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.795224 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.795469 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.795470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.795470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.795578 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:06.795631 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.796067 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.796148 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:07:06.796214 392 external_mini_cluster.cc:1428] Setting key 922726e0f8cf574f6b95f7792fa3a7f0
I20260924 23:07:06.796499 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.786877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36147 (local address 127.0.98.3:46267)
0924 23:07:06.787075 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:06.787114 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:06.787708 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:06.788130 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:06.788134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:06.788767 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:06.788992 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.788997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.789718 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.789719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.790609 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.790611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.791028 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.791028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.791126 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.791698 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:06.791707 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:06.793131 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:06.794777 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:06.794783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:06.794785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:06.795030 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:06.795293 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:06.795294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:06.795294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:06.795403 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:06.795638 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:06.795643 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:06.795859 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 23:07:06.795999 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 23:07:06.796229 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":90,"threads_started":1}
I20260924 23:07:06.796994 11067 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36147
I20260924 23:07:06.797112 11067 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:06.797309 11067 heartbeater.cc:511] Master 127.0.98.62:36147 requested a full tablet report, sending...
I20260924 23:07:06.797745 10607 ts_manager.cc:194] Registered new tserver with Master: 55a4405ef6ed46c78dcbae099024f0d1 (127.0.98.3:45999)
I20260924 23:07:06.798269 10607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46267
I20260924 23:07:06.808223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:07:06.816095 10654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.809616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.0.98.62:36147)
0924 23:07:06.809687 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.809692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.809754 (+ 62us) server_negotiation.cc:407] Connection header received
0924 23:07:06.809813 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.809814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.809822 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.809928 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:06.810524 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.811065 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.811841 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.812068 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.812129 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.813465 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:06.813467 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:06.813467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:06.813475 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:06.815035 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.815379 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.815380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.815380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.815432 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:06.815599 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:06.815600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:06.815600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:06.815743 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:06.815817 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.815961 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.816027 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:07:06.818048 10607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:07:06.819585 10607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:07:06.826134 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.821830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.0.98.1:42875)
0924 23:07:06.822254 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.822259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.822390 (+ 131us) server_negotiation.cc:407] Connection header received
0924 23:07:06.822459 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.822461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.822477 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.822555 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:06.822890 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.823981 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.825165 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.825702 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.825775 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.825830 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.825999 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.826043 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":101,"threads_started":1}
I20260924 23:07:06.826151 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.821755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42875 (local address 127.0.0.1:48888)
0924 23:07:06.822308 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:06.822316 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:06.822391 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:06.822791 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:06.822791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:06.822793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:06.822831 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.822833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.824091 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.824092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.825072 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.825074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.825776 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.825777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.825869 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.825898 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 23:07:06.825976 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 23:07:06.826059 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":294,"thread_start_us":72,"threads_started":1}
I20260924 23:07:06.826233 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.822040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.98.2:41371)
0924 23:07:06.822241 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.822245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.822257 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:07:06.822296 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.822298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.822310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.822394 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:06.822591 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.823537 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.824838 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.825557 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.825629 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.825685 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.826106 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.826154 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260924 23:07:06.826320 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.821755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45999 (local address 127.0.0.1:57350)
0924 23:07:06.822120 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:06.822130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:06.822174 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:06.822693 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:06.822694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:06.822697 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:06.822748 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.822750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.824106 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.824107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.825139 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.825141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.826097 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.826098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.826156 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.826160 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:07:06.826211 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:07:06.826244 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260924 23:07:06.826433 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.821755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41371 (local address 127.0.0.1:56210)
0924 23:07:06.821899 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:06.821945 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:06.822012 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:06.822413 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:06.822416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:06.822419 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:06.822494 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.822496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.823625 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.823626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.824752 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:06.824755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.825766 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.825767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:06.825849 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.825863 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:07:06.825975 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 23:07:06.826048 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260924 23:07:06.826591 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:06.822048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.0.98.3:45999)
0924 23:07:06.822548 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:06.822552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:06.822562 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:07:06.822602 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:06.822604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:06.822619 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:06.822681 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:06.822825 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.823981 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.825229 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:06.825944 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:06.826003 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:06.826065 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:06.826434 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 23:07:06.826478 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":84,"threads_started":1}
I20260924 23:07:06.827303 11001 tablet_service.cc:1467] Processing CreateTablet for tablet 4a840d55da324d7ba0d74bef5b93d89f (DEFAULT_TABLE table=test-table [id=2a953594aa9f43b782effba020b24ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:07:06.827394 10733 tablet_service.cc:1467] Processing CreateTablet for tablet 4a840d55da324d7ba0d74bef5b93d89f (DEFAULT_TABLE table=test-table [id=2a953594aa9f43b782effba020b24ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:07:06.827577 11001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a840d55da324d7ba0d74bef5b93d89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:06.827649 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a840d55da324d7ba0d74bef5b93d89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:06.830188 11084 tablet_bootstrap.cc:492] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Bootstrap starting.
I20260924 23:07:06.830226 10867 tablet_service.cc:1467] Processing CreateTablet for tablet 4a840d55da324d7ba0d74bef5b93d89f (DEFAULT_TABLE table=test-table [id=2a953594aa9f43b782effba020b24ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:07:06.830592 10867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a840d55da324d7ba0d74bef5b93d89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:06.830862 11084 tablet_bootstrap.cc:654] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:06.831434 11084 log.cc:824] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:06.831773 11085 tablet_bootstrap.cc:492] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: Bootstrap starting.
I20260924 23:07:06.832156 11084 tablet_bootstrap.cc:492] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: No bootstrap required, opened a new log
I20260924 23:07:06.832203 11084 ts_tablet_manager.cc:1397] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:07:06.832419 11085 tablet_bootstrap.cc:654] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:06.832947 11085 log.cc:824] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:06.833642 11085 tablet_bootstrap.cc:492] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: No bootstrap required, opened a new log
I20260924 23:07:06.833698 11085 ts_tablet_manager.cc:1397] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:07:06.833748 11084 raft_consensus.cc:348] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:06.833927 11084 raft_consensus.cc:374] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:06.833971 11084 raft_consensus.cc:729] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a4405ef6ed46c78dcbae099024f0d1, State: Initialized, Role: FOLLOWER
I20260924 23:07:06.834064 11088 tablet_bootstrap.cc:492] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: Bootstrap starting.
I20260924 23:07:06.834101 11084 consensus_queue.cc:260] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:06.834381 11084 ts_tablet_manager.cc:1428] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:07:06.834619 11067 heartbeater.cc:503] Master 127.0.98.62:36147 was elected leader, sending a full tablet report...
I20260924 23:07:06.834794 11088 tablet_bootstrap.cc:654] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:06.835273 11088 log.cc:824] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:06.835767 11085 raft_consensus.cc:348] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:06.835949 11085 raft_consensus.cc:374] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:06.835994 11085 raft_consensus.cc:729] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61c1687d28d41ef86fabf62da32f9c3, State: Initialized, Role: FOLLOWER
I20260924 23:07:06.836052 11088 tablet_bootstrap.cc:492] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: No bootstrap required, opened a new log
I20260924 23:07:06.836107 11088 ts_tablet_manager.cc:1397] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:07:06.836129 11085 consensus_queue.cc:260] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:06.836385 11085 ts_tablet_manager.cc:1428] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:07:06.836426 10799 heartbeater.cc:503] Master 127.0.98.62:36147 was elected leader, sending a full tablet report...
I20260924 23:07:06.838066 11088 raft_consensus.cc:348] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:06.838217 11088 raft_consensus.cc:374] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:06.838240 11088 raft_consensus.cc:729] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f4bbc048e5404f824c6f1077192ff0, State: Initialized, Role: FOLLOWER
I20260924 23:07:06.838333 11088 consensus_queue.cc:260] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:06.838523 11088 ts_tablet_manager.cc:1428] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:07:06.838587 10933 heartbeater.cc:503] Master 127.0.98.62:36147 was elected leader, sending a full tablet report...
W20260924 23:07:06.882018 10934 tablet.cc:2369] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:07:06.961269 10800 tablet.cc:2369] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:07:07.001938 11089 raft_consensus.cc:482] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:07:07.002144 11089 raft_consensus.cc:504] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:07.002617 11089 leader_election.cc:288] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f61c1687d28d41ef86fabf62da32f9c3 (127.0.98.1:42875), 06f4bbc048e5404f824c6f1077192ff0 (127.0.98.2:41371)
I20260924 23:07:07.006963 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.002800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41371 (local address 127.0.98.3:47163)
0924 23:07:07.002909 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:07.002922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:07.002981 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:07.003511 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:07.003512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:07.003513 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:07.003559 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:07.003560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.004495 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.004497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:07.005847 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:07.005860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.006675 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.006676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:07.006747 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:07.006751 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:07:07.006836 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 23:07:07.006882 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 23:07:07.006963 11082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.003078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47163 (local address 127.0.98.2:41371)
0924 23:07:07.003335 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:07.003337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:07.003350 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:07:07.003398 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:07.003399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:07.003414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:07.003498 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:07.003632 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.004392 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.006011 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.006586 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.006666 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:07.006678 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:07.006843 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 23:07:07.006890 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260924 23:07:07.007493 10887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a840d55da324d7ba0d74bef5b93d89f" candidate_uuid: "55a4405ef6ed46c78dcbae099024f0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4bbc048e5404f824c6f1077192ff0" is_pre_election: true
I20260924 23:07:07.007668 10887 raft_consensus.cc:2454] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a4405ef6ed46c78dcbae099024f0d1 in term 0.
I20260924 23:07:07.007961 10956 leader_election.cc:302] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f4bbc048e5404f824c6f1077192ff0, 55a4405ef6ed46c78dcbae099024f0d1; no voters:
I20260924 23:07:07.008123 11089 raft_consensus.cc:2781] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:07:07.008176 11089 raft_consensus.cc:482] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:07:07.008201 11089 raft_consensus.cc:3037] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:07.008477 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.002800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42875 (local address 127.0.98.3:55095)
0924 23:07:07.003154 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:07.003163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:07.003209 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:07.003374 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:07.003375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:07.003376 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:07.003462 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:07.003464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.005942 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.005944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:07.007149 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:07.007152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.008141 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.008142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:07.008244 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:07.008249 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:07:07.008341 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 23:07:07.008390 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":12,"thread_start_us":100,"threads_started":1}
I20260924 23:07:07.008486 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.003046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55095 (local address 127.0.98.1:42875)
0924 23:07:07.003096 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:07.003098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:07.003267 (+ 169us) server_negotiation.cc:407] Connection header received
0924 23:07:07.003295 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:07.003296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:07.003307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:07.003359 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:07.003644 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.004392 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.007333 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.007985 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.008082 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:07.008095 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:07.008357 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 23:07:07.008404 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:07:07.008929 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a840d55da324d7ba0d74bef5b93d89f" candidate_uuid: "55a4405ef6ed46c78dcbae099024f0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61c1687d28d41ef86fabf62da32f9c3" is_pre_election: true
I20260924 23:07:07.009099 10753 raft_consensus.cc:2454] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55a4405ef6ed46c78dcbae099024f0d1 in term 0.
I20260924 23:07:07.009218 11089 raft_consensus.cc:504] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:07.009366 11089 leader_election.cc:288] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [CANDIDATE]: Term 1 election: Requested vote from peers f61c1687d28d41ef86fabf62da32f9c3 (127.0.98.1:42875), 06f4bbc048e5404f824c6f1077192ff0 (127.0.98.2:41371)
I20260924 23:07:07.009544 10887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a840d55da324d7ba0d74bef5b93d89f" candidate_uuid: "55a4405ef6ed46c78dcbae099024f0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f4bbc048e5404f824c6f1077192ff0"
I20260924 23:07:07.009615 10753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a840d55da324d7ba0d74bef5b93d89f" candidate_uuid: "55a4405ef6ed46c78dcbae099024f0d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61c1687d28d41ef86fabf62da32f9c3"
I20260924 23:07:07.009631 10887 raft_consensus.cc:3037] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:07.009709 10753 raft_consensus.cc:3037] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:07.010478 10887 raft_consensus.cc:2454] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a4405ef6ed46c78dcbae099024f0d1 in term 1.
I20260924 23:07:07.010488 10753 raft_consensus.cc:2454] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55a4405ef6ed46c78dcbae099024f0d1 in term 1.
I20260924 23:07:07.010650 10956 leader_election.cc:302] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f4bbc048e5404f824c6f1077192ff0, 55a4405ef6ed46c78dcbae099024f0d1; no voters:
I20260924 23:07:07.010746 11089 raft_consensus.cc:2781] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:07:07.010882 11089 raft_consensus.cc:686] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 1 LEADER]: Becoming Leader. State: Replica: 55a4405ef6ed46c78dcbae099024f0d1, State: Running, Role: LEADER
I20260924 23:07:07.010985 11089 consensus_queue.cc:237] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } }
I20260924 23:07:07.011861 10607 catalog_manager.cc:5495] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55a4405ef6ed46c78dcbae099024f0d1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "55a4405ef6ed46c78dcbae099024f0d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a4405ef6ed46c78dcbae099024f0d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45999 } health_report { overall_health: HEALTHY } } }
W20260924 23:07:07.036746 11068 tablet.cc:2369] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:07:07.056449 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.053792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.98.3:45999)
0924 23:07:07.053869 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:07.053871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:07.053930 (+ 59us) server_negotiation.cc:407] Connection header received
0924 23:07:07.054014 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:07.054015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:07.054035 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:07.054098 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:07:07.054263 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.054800 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.055617 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:07.055819 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:07.055879 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:07.056021 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:07:07.056137 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:07:07.056210 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:07.056257 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 23:07:07.056277 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:07:07.058535 10753 raft_consensus.cc:1261] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 1 FOLLOWER]: Refusing update from remote peer 55a4405ef6ed46c78dcbae099024f0d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:07:07.058538 10887 raft_consensus.cc:1261] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 1 FOLLOWER]: Refusing update from remote peer 55a4405ef6ed46c78dcbae099024f0d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:07:07.058955 11094 consensus_queue.cc:1035] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f4bbc048e5404f824c6f1077192ff0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:07:07.059118 11089 consensus_queue.cc:1035] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61c1687d28d41ef86fabf62da32f9c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:07:07.063988 10608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:07:07.064100 10608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:07:07.064865 10608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a3206d6e8b44e8886de8d1f454e171e: Sending DeleteTablet for 3 replicas of tablet 4a840d55da324d7ba0d74bef5b93d89f
I20260924 23:07:07.065228 10733 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a840d55da324d7ba0d74bef5b93d89f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:07:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48888
I20260924 23:07:07.065235 10867 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a840d55da324d7ba0d74bef5b93d89f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:07:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56210
I20260924 23:07:07.065459 11107 tablet_replica.cc:330] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3: stopping tablet replica
I20260924 23:07:07.065459 11001 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a840d55da324d7ba0d74bef5b93d89f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:07:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57350
I20260924 23:07:07.065534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10670
I20260924 23:07:07.065593 11107 raft_consensus.cc:2229] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:07:07.065702 11107 raft_consensus.cc:2258] T 4a840d55da324d7ba0d74bef5b93d89f P f61c1687d28d41ef86fabf62da32f9c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:07:07.067312 11109 tablet_replica.cc:330] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: stopping tablet replica
I20260924 23:07:07.067418 11109 raft_consensus.cc:2229] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:07:07.067540 11109 raft_consensus.cc:2258] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:07:07.067978 11109 ts_tablet_manager.cc:1905] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:07:07.070294 11109 ts_tablet_manager.cc:1918] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:07:07.070375 11109 log.cc:1196] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/4a840d55da324d7ba0d74bef5b93d89f
I20260924 23:07:07.070698 11109 ts_tablet_manager.cc:1939] T 4a840d55da324d7ba0d74bef5b93d89f P 55a4405ef6ed46c78dcbae099024f0d1: Deleting consensus metadata
I20260924 23:07:07.071132 10593 catalog_manager.cc:4841] TS 55a4405ef6ed46c78dcbae099024f0d1 (127.0.98.3:45999): tablet 4a840d55da324d7ba0d74bef5b93d89f (table test-table [id=2a953594aa9f43b782effba020b24ff5]) successfully deleted
W20260924 23:07:07.072124 10595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42875 (error 108)
I20260924 23:07:07.072651 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.072529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42875 (local address 127.0.0.1:48904)
0924 23:07:07.072578 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:07.072617 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:07:07.072664 11108 tablet_replica.cc:330] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0: stopping tablet replica
I20260924 23:07:07.072769 11108 raft_consensus.cc:2229] T 4a840d55da324d7ba0d74bef5b93d89f P 06f4bbc048e5404f824c6f1077192ff0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 23:07:07.072772 10595 catalog_manager.cc:4584] TS f61c1687d28d41ef86fabf62da32f9c3 (127.0.98.1:42875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a840d55da324d7ba0d74bef5b93d89f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42875: connect: Connection refused (error 111)
I20260924 23:07:07.072944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10804
I20260924 23:07:07.077821 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:07.077694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41371 (local address 127.0.0.1:56224)
0924 23:07:07.077743 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:07.077783 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 23:07:07.078069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10938
I20260924 23:07:07.083428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 10576
2026-09-24T23:07:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2135 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:07:07 dist-test-slave-ph9j krb5kdc[11113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:07:07 dist-test-slave-ph9j krb5kdc[11113](info): set up 2 sockets
Sep 24 23:07:07 dist-test-slave-ph9j krb5kdc[11113](info): commencing operation
krb5kdc: starting...
W20260924 23:07:08.194751 392 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:07:08 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:07:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 23:07:08.365314 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKo6W1P/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/postgres -l logfile start
2026-09-24 23:07:11.007 UTC [11143] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 23:07:11.007 UTC [11143] LOG: listening on IPv4 address "127.0.98.20", port 40239
2026-09-24 23:07:11.011 UTC [11143] LOG: listening on Unix socket "/tmp/.s.PGSQL.40239"
2026-09-24 23:07:11.034 UTC [11146] LOG: database system was shut down at 2026-09-24 23:07:09 UTC
2026-09-24 23:07:11.039 UTC [11143] LOG: database system is ready to accept connections
I20260924 23:07:11.994865 392 mini_postgres.cc:96] Postgres bound to 40239
2026-09-24 23:07:12.001 UTC [11154] FATAL: database "slave" does not exist
127.0.98.20:40239 - accepting connections
I20260924 23:07:12.001713 392 mini_ranger.cc:161] Starting Ranger...
I20260924 23:07:12.021426 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 23:07:12.534524 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 23:07:12.534626 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-admin
2026-09-24 23:07:12,824 [I] DB FLAVOR :POSTGRES
2026-09-24 23:07:12,825 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:07:12,825 [I] Checking connection..
2026-09-24 23:07:12,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 23:07:13,093 [I] Checking connection passed.
2026-09-24 23:07:13,093 [I] --------- Verifying version history table ---------
2026-09-24 23:07:13,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:07:13,334 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:07:13,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:07:13,570 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:07:13,571 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 23:07:13,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 23:07:13.797 UTC [11218] WARNING: there is no transaction in progress
2026-09-24 23:07:13,803 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 23:07:13,804 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:07:14,038 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 23:07:14,038 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:07:14,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 23:07:14,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 23:07:14,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ph9j.c.gcp-upstream.internal','N') ;"
2026-09-24 23:07:14,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:07:15,013 [I] Table x_portal_user does not exist in database ranger
2026-09-24 23:07:15,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:07:15,251 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 23:07:15,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:07:15,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:07:15,762 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 23:07:15,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 23:07:16.757 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.775 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.794 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.812 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.829 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.847 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.905 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.913 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.928 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.938 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.949 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.963 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.973 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.980 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.987 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:16.993 UTC [11326] WARNING: there is no transaction in progress
2026-09-24 23:07:17,724 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 23:07:17,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 23:07:17,983 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 23:07:17,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:07:18,231 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 23:07:18,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:07:18,487 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 23:07:18,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:07:18,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:07:18,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ph9j.c.gcp-upstream.internal';"
2026-09-24 23:07:19,207 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 23:07:19,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 23:07:19,469 [I] DB_PATCHES have already been applied
I20260924 23:07:19.475044 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKo6W1P/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 23:07:19.475183 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 23:07:19.478328 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40707
Sep 24, 2026 11:07:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:07:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:07:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:07:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:07:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:07:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:07:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 23:07:20.595659 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.594s user 0.000s sys 0.009s
Sep 24, 2026 11:07:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:07:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 11:07:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:07:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:07:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 23:07:41.276274 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 23:07:41 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 23:07:41.364512 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 23:07:41.374873 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 23:07:42.304095 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 23:07:42.304180 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms
I20260924 23:07:42.304199 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:40239
2026-09-24 23:07:42,480 [I] DB FLAVOR :POSTGRES
2026-09-24 23:07:42,480 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:07:42,480 [I] Checking connection
2026-09-24 23:07:42,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 23:07:42,748 [I] connection success
2026-09-24 23:07:42,748 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 23:07:42,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 23:07:42,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 23:07:42,980 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 23:07:42,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:07:42,980 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 23:07:42,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40239/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 23:07:43,241 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 23:07:43.723193 392 mini_ranger_kms.cc:310] Created kms service
I20260924 23:07:43.892390 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 23:07:43.937177 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 23:07:43 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291261, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:07:44.324698 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 23:07:44.324831 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 23:07:44.324853 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 23:07:44.331132 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11506
I20260924 23:07:44.331301 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58539
Sep 24, 2026 11:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:07:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 23:07:46.708732 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.344s user 0.000s sys 0.158s
I20260924 23:07:46.708834 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 23:07:46.708907 392 mini_ranger_kms.cc:401] 127.0.98.20:58539/kms/v1/keys
Sep 24 23:07:47 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 23:07:47 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 23:07:47 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 23:07:50 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:07:50.181115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42509
--ranger_config_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58539/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:50.279177 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:50.279407 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:50.279428 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:50.281013 11557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:07:50.281061 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:50.281075 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:50.281088 11557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:07:50.281100 11557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:07:50.282755 11557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58539/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42509
--ranger_config_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:50.283015 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:50.283250 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:50.289294 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:50.289418 11557 server_base.cc:1034] running on GCE node
W20260924 23:07:50.289348 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:50.289311 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:50.289919 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:50.290266 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:50.291445 11557 hybrid_clock.cc:648] HybridClock initialized: now 1790291270291431 us; error 32 us; skew 500 ppm
Sep 24 23:07:50 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:50.294090 11557 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:07:50.294615 11557 webserver.cc:460] Webserver started at http://127.0.98.62:40255/ using document root <none> and password file <none>
I20260924 23:07:50.294855 11557 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:50.294895 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:50.295028 11557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:07:50 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:07:50.359927 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "c135bb5a52be462782808d9d3e8bb589"
format_stamp: "Formatted at 2026-09-24 23:07:50 on dist-test-slave-ph9j"
server_key: "77e29cddd3427ff82ffd364ba701ffcd"
server_key_iv: "a2ad2a20b2a26ddde278b536564c4642"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:50.360428 11557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "c135bb5a52be462782808d9d3e8bb589"
format_stamp: "Formatted at 2026-09-24 23:07:50 on dist-test-slave-ph9j"
server_key: "77e29cddd3427ff82ffd364ba701ffcd"
server_key_iv: "a2ad2a20b2a26ddde278b536564c4642"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:50.361964 11557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 23:07:50.377736 11572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:50.377954 11557 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:50.378062 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "c135bb5a52be462782808d9d3e8bb589"
format_stamp: "Formatted at 2026-09-24 23:07:50 on dist-test-slave-ph9j"
server_key: "77e29cddd3427ff82ffd364ba701ffcd"
server_key_iv: "a2ad2a20b2a26ddde278b536564c4642"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:50.378172 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:50.399041 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:50.401221 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:50.401396 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:50.406388 11557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42509
I20260924 23:07:50.406409 11624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42509 every 8 connection(s)
I20260924 23:07:50.406804 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:07:50.407902 11625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:50.408069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 11557
I20260924 23:07:50.408169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:07:50.408371 392 external_mini_cluster.cc:1428] Setting key 5dc8b6f7f96855d205d71c618d2bd5e7
I20260924 23:07:50.411000 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589: Bootstrap starting.
I20260924 23:07:50.411628 11625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:50.411921 11625 log.cc:824] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:50.412650 11625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589: No bootstrap required, opened a new log
I20260924 23:07:50.415184 11625 raft_consensus.cc:348] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135bb5a52be462782808d9d3e8bb589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 23:07:50.415398 11625 raft_consensus.cc:374] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:50.415438 11625 raft_consensus.cc:729] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c135bb5a52be462782808d9d3e8bb589, State: Initialized, Role: FOLLOWER
I20260924 23:07:50.415629 11625 consensus_queue.cc:260] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135bb5a52be462782808d9d3e8bb589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 23:07:50.415707 11625 raft_consensus.cc:388] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:07:50.415737 11625 raft_consensus.cc:482] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:07:50.415809 11625 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:50.416415 11625 raft_consensus.cc:504] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135bb5a52be462782808d9d3e8bb589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 23:07:50.416540 11625 leader_election.cc:302] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c135bb5a52be462782808d9d3e8bb589; no voters:
Sep 24 23:07:50 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:50.416700 11625 leader_election.cc:288] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:07:50.416888 11631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:07:50.417001 11625 sys_catalog.cc:564] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:07:50.417094 11631 raft_consensus.cc:686] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [term 1 LEADER]: Becoming Leader. State: Replica: c135bb5a52be462782808d9d3e8bb589, State: Running, Role: LEADER
I20260924 23:07:50.417280 11631 consensus_queue.cc:237] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135bb5a52be462782808d9d3e8bb589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } }
I20260924 23:07:50.417701 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c135bb5a52be462782808d9d3e8bb589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135bb5a52be462782808d9d3e8bb589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } } }
I20260924 23:07:50.417860 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:50.418068 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c135bb5a52be462782808d9d3e8bb589. Latest consensus state: current_term: 1 leader_uuid: "c135bb5a52be462782808d9d3e8bb589" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135bb5a52be462782808d9d3e8bb589" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42509 } } }
I20260924 23:07:50.418200 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589 [sys.catalog]: This master's current role is: LEADER
I20260924 23:07:50.418356 11640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:07:50.418488 11625 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 23:07:50.418903 11640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:07:50.421912 11647 mvcc.cc:205] Tried to move back new op lower bound from 7333033043639246848 to 7333033043629809664. Current Snapshot: MvccSnapshot[applied={T|T < 7333033043639246848}]
I20260924 23:07:50.424124 11640 catalog_manager.cc:1347] Generated new cluster ID: 782e3a26a3474721ae5c61a2326db187
I20260924 23:07:50.424203 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:50.408948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.0.98.62:42509)
0924 23:07:50.409163 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:50.409174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:50.409214 (+ 40us) server_negotiation.cc:407] Connection header received
0924 23:07:50.409783 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:50.409794 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:50.409865 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:50.410064 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:50.410856 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:50.412286 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:50.413443 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:50.414107 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:50.414224 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:50.417192 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:50.417196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:50.417196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:50.417209 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:50.419206 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:50.419627 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:50.419630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:50.419630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:50.419726 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:50.419921 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:50.419922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:50.419922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:50.421651 (+ 1729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:50.423012 (+ 1361us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:50.423250 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 23:07:50.423860 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260924 23:07:50.424214 11640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:07:50.432655 11640 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:07:50.433470 11640 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:07:50.436595 11640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589: Generated new TSK 0
I20260924 23:07:50.436815 11640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:07:51.153191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58539/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:51.242635 11670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:51.242884 11670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:51.242924 11670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:51.244102 11670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:51.244171 11670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:51.244220 11670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:07:51.245821 11670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58539/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:51.246105 11670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:51.246357 11670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:51.252012 11676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:51.252024 11675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:51.252319 11678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:51.252668 11670 server_base.cc:1034] running on GCE node
I20260924 23:07:51.252914 11670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:51.253203 11670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:51.254364 11670 hybrid_clock.cc:648] HybridClock initialized: now 1790291271254341 us; error 37 us; skew 500 ppm
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:51.256868 11670 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:07:51.257378 11670 webserver.cc:460] Webserver started at http://127.0.98.1:44961/ using document root <none> and password file <none>
I20260924 23:07:51.257651 11670 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:51.257714 11670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:51.257840 11670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:07:51.271221 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "692c77d1cb9047f6b021cf7c09a09470"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "3724de3495aadbb47265b4f99e0d4b81"
server_key_iv: "f1838fa397d9810560e770fe5fed9550"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.271787 11670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "692c77d1cb9047f6b021cf7c09a09470"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "3724de3495aadbb47265b4f99e0d4b81"
server_key_iv: "f1838fa397d9810560e770fe5fed9550"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.273819 11670 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 23:07:51.288746 11685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.288998 11670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:51.289115 11670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "692c77d1cb9047f6b021cf7c09a09470"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "3724de3495aadbb47265b4f99e0d4b81"
server_key_iv: "f1838fa397d9810560e770fe5fed9550"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.289233 11670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:51.307752 11670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:51.310297 11670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:51.310459 11670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:51.310770 11670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:51.311167 11670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:51.311229 11670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.311265 11670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:51.311306 11670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.319087 11670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40863
I20260924 23:07:51.319523 11670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:07:51.320022 11800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40863 every 8 connection(s)
I20260924 23:07:51.322661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 11670
I20260924 23:07:51.322793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:07:51.322976 392 external_mini_cluster.cc:1428] Setting key 1d0ef41ebf80f19e584f9ed3b42761ab
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:51.332340 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.322273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58013 (local address 127.0.98.62:42509)
0924 23:07:51.323526 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.323528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.323534 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:07:51.323556 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.323557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.323568 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.324459 (+ 891us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:51.324500 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.325278 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.326814 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.327032 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.327126 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.329268 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:51.329270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:51.329270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:51.329278 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:51.330901 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.331315 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.331316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.331317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.331382 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.331649 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.331650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.331651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.331777 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:51.331853 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.332140 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.332237 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":82,"threads_started":1}
I20260924 23:07:51.332542 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.322268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.1:58013)
0924 23:07:51.322431 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:51.322474 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:51.323347 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:51.323744 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:51.323748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:51.324018 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:51.324364 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.324371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.325385 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.325386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.326700 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.326703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.327135 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.327136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.327277 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.327848 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:51.327856 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:51.329148 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:51.331005 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:51.331012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:51.331014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:51.331236 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:51.331459 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:51.331460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:51.331461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:51.331576 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:51.331844 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:51.331850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:51.332006 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 23:07:51.332122 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 23:07:51.332285 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 23:07:51.333127 11801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42509
I20260924 23:07:51.333257 11801 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:51.333449 11801 heartbeater.cc:511] Master 127.0.98.62:42509 requested a full tablet report, sending...
I20260924 23:07:51.334043 11589 ts_manager.cc:194] Registered new tserver with Master: 692c77d1cb9047f6b021cf7c09a09470 (127.0.98.1:40863)
I20260924 23:07:51.334955 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58013
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:07:51.357755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58539/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:51.476282 11807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:51.476547 11807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:51.476593 11807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:51.478070 11807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:51.478157 11807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:51.478207 11807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:07:51.479990 11807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58539/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:51.480291 11807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:51.480614 11807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:51.486685 11815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:51.487262 11814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:51.486685 11817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:51.488333 11807 server_base.cc:1034] running on GCE node
I20260924 23:07:51.488567 11807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:51.488867 11807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:51.490027 11807 hybrid_clock.cc:648] HybridClock initialized: now 1790291271490013 us; error 30 us; skew 500 ppm
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:51.493027 11807 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:07:51.493674 11807 webserver.cc:460] Webserver started at http://127.0.98.2:42287/ using document root <none> and password file <none>
I20260924 23:07:51.493983 11807 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:51.494074 11807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:51.494233 11807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:51.507356 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "f464f495fbe64589987021e6ef2fffb8"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "c2966efd102b09821b5ac25c827b6702"
server_key_iv: "acd7254a58838b12cf3a6e1369b4be8b"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.507949 11807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f464f495fbe64589987021e6ef2fffb8"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "c2966efd102b09821b5ac25c827b6702"
server_key_iv: "acd7254a58838b12cf3a6e1369b4be8b"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.516892 11807 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260924 23:07:51.538059 11824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.538363 11807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 23:07:51.538462 11807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "f464f495fbe64589987021e6ef2fffb8"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "c2966efd102b09821b5ac25c827b6702"
server_key_iv: "acd7254a58838b12cf3a6e1369b4be8b"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.538564 11807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:51.563608 11807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:51.566097 11807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:51.566254 11807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:51.566485 11807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:51.566851 11807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:51.566895 11807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.566928 11807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:51.566954 11807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.575058 11807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36665
I20260924 23:07:51.575524 11807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:07:51.576802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 11807
I20260924 23:07:51.576901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:07:51.577071 392 external_mini_cluster.cc:1428] Setting key e8bc44d73a0123a83170e876a8514d28
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:51.586190 11937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36665 every 8 connection(s)
I20260924 23:07:51.587736 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.576293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34213 (local address 127.0.98.62:42509)
0924 23:07:51.576347 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.576349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.576598 (+ 249us) server_negotiation.cc:407] Connection header received
0924 23:07:51.577305 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.577306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.577318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.577354 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:51.578286 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.579151 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.580774 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.581104 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.581184 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.583781 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:51.583783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:51.583783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:51.583793 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:51.585763 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.586470 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.586471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.586472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.586537 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.586865 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.586866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.586866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.587018 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:51.587078 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.587517 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.587630 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:07:51.588568 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.576260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.2:34213)
0924 23:07:51.576437 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:51.576487 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:51.577166 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:51.577558 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:51.577563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:51.577844 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:51.578163 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.578170 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.579274 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.579276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.580686 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.580688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.581203 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.581204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.581361 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.581992 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:51.582003 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:51.583674 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:51.585901 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:51.585910 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:51.585913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:51.586370 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:51.586620 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:51.586621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:51.586621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:51.586774 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:51.587114 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:51.587121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:51.587335 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 23:07:51.588109 (+ 774us) client_negotiation.cc:240] Negotiation successful
0924 23:07:51.588309 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260924 23:07:51.589335 11938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42509
I20260924 23:07:51.589496 11938 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:51.589763 11938 heartbeater.cc:511] Master 127.0.98.62:42509 requested a full tablet report, sending...
I20260924 23:07:51.590209 11589 ts_manager.cc:194] Registered new tserver with Master: f464f495fbe64589987021e6ef2fffb8 (127.0.98.2:36665)
I20260924 23:07:51.590651 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34213
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:07:51.638576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58539/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:07:51.768409 11944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:07:51.768813 11944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:07:51.768908 11944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:07:51.770984 11944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:07:51.771122 11944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:07:51.771209 11944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:07:51.773675 11944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58539/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:07:51.774308 11944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:07:51.774682 11944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:07:51.781504 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:51.781507 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:07:51.782717 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:07:51.783703 11944 server_base.cc:1034] running on GCE node
I20260924 23:07:51.784114 11944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:07:51.784425 11944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:07:51.785612 11944 hybrid_clock.cc:648] HybridClock initialized: now 1790291271785583 us; error 47 us; skew 500 ppm
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:07:51.789177 11944 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:07:51.789738 11944 webserver.cc:460] Webserver started at http://127.0.98.3:46039/ using document root <none> and password file <none>
I20260924 23:07:51.790041 11944 fs_manager.cc:359] Metadata directory not provided
I20260924 23:07:51.790097 11944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:07:51.790231 11944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:07:51.807205 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "df0ba32419874c12baa77444c841096b"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "ea6c4e11e434eda7dbe3b36394fa586f"
server_key_iv: "09552f001483f02c6ab8b25c2d398fbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.807853 11944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df0ba32419874c12baa77444c841096b"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "ea6c4e11e434eda7dbe3b36394fa586f"
server_key_iv: "09552f001483f02c6ab8b25c2d398fbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.810731 11944 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 23:07:51.828450 11959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.828655 11944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:07:51.828745 11944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "df0ba32419874c12baa77444c841096b"
format_stamp: "Formatted at 2026-09-24 23:07:51 on dist-test-slave-ph9j"
server_key: "ea6c4e11e434eda7dbe3b36394fa586f"
server_key_iv: "09552f001483f02c6ab8b25c2d398fbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:07:51.828858 11944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:07:51.867271 11944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:07:51.869709 11944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:07:51.869866 11944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:07:51.870124 11944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:07:51.870523 11944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:07:51.870571 11944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.870602 11944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:07:51.870626 11944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:07:51.880303 11944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41021
I20260924 23:07:51.880445 12072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41021 every 8 connection(s)
I20260924 23:07:51.880862 11944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:07:51 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:07:51.889230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 11944
I20260924 23:07:51.889343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:07:51.889523 392 external_mini_cluster.cc:1428] Setting key c046643bce1ec78df1c99949bed07245
I20260924 23:07:51.892222 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.881666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33879 (local address 127.0.98.62:42509)
0924 23:07:51.881722 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.881725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.882439 (+ 714us) server_negotiation.cc:407] Connection header received
0924 23:07:51.883438 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.883440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.883454 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.883490 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:51.884392 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.885011 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.886079 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.886344 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.886393 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.889019 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:51.889020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:51.889021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:51.889028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:51.890759 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.891215 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.891216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.891216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.891268 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.891545 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.891546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.891546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.891664 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:51.891704 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.892056 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.892134 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:07:51.892926 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.881633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42509 (local address 127.0.98.3:33879)
0924 23:07:51.881820 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:51.881936 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:51.883264 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:51.883699 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:51.883704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:51.883985 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:07:51.884266 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.884272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.885105 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.885105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.885980 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.885983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.886490 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.886491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.886642 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.887245 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:07:51.887256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:07:51.888898 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:07:51.890855 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:51.890862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:51.890864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:51.891134 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:51.891350 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:07:51.891350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:07:51.891351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:07:51.891478 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:07:51.891744 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:07:51.891748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:07:51.891915 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 23:07:51.892557 (+ 642us) client_negotiation.cc:240] Negotiation successful
0924 23:07:51.892712 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":114,"threads_started":1}
I20260924 23:07:51.893554 12073 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42509
I20260924 23:07:51.893684 12073 heartbeater.cc:464] Registering TS with master...
I20260924 23:07:51.893932 12073 heartbeater.cc:511] Master 127.0.98.62:42509 requested a full tablet report, sending...
I20260924 23:07:51.894320 11589 ts_manager.cc:194] Registered new tserver with Master: df0ba32419874c12baa77444c841096b (127.0.98.3:41021)
I20260924 23:07:51.894784 11589 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33879
I20260924 23:07:51.901067 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:07:51.916438 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.902913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.0.98.62:42509)
0924 23:07:51.902954 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.902956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.903280 (+ 324us) server_negotiation.cc:407] Connection header received
0924 23:07:51.903372 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.903373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.903383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.903411 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:07:51.904603 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.905386 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.911466 (+ 6080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.911757 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.911815 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.913500 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:07:51.913502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:07:51.913502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:07:51.913510 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:07:51.915215 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.915609 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.915610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.915610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.915664 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:07:51.915867 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:07:51.915868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:07:51.915868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:07:51.915992 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:07:51.916062 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.916261 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.916339 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:07:51.918632 11589 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:07:51.920897 11589 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:07:51.931459 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.923979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41021 (local address 127.0.0.1:49240)
0924 23:07:51.925135 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:51.925175 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:51.925240 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:51.925530 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:51.925533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:51.925535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:51.925619 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.925620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.927509 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.927510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.928763 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.928766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.931149 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.931150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.931235 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.931250 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:07:51.931325 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:07:51.931378 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":66,"threads_started":1}
I20260924 23:07:51.931907 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.924418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.0.98.3:41021)
0924 23:07:51.925075 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.925080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.925319 (+ 239us) server_negotiation.cc:407] Connection header received
0924 23:07:51.925362 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.925364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.925380 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.925445 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:51.925699 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.927431 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.928888 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.929606 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.929676 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.929757 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.931794 (+ 2037us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.931833 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":62,"threads_started":1}
I20260924 23:07:51.934623 12001 tablet_service.cc:1467] Processing CreateTablet for tablet b17566152bc64a55bb1d42288ffda05c (DEFAULT_TABLE table=test-table [id=34509dede017425eb02c818c77080b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:07:51.935014 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17566152bc64a55bb1d42288ffda05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:51.941816 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.924632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40863 (local address 127.0.0.1:40868)
0924 23:07:51.927034 (+ 2402us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:51.927042 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:51.927082 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:51.928002 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:51.928003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:51.928005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:51.928056 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.928058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.931138 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.931139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.932724 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.932726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.941552 (+ 8826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.941554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.941631 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.941637 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:07:51.941668 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 23:07:51.941717 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2390,"thread_start_us":66,"threads_started":1}
I20260924 23:07:51.942821 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.924204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36665 (local address 127.0.0.1:50784)
0924 23:07:51.926910 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:51.926920 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:51.926962 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:51.928135 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:51.928135 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:51.928137 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:51.928176 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.928177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.932833 (+ 4656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.932833 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.939485 (+ 6652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:51.939490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.942570 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.942571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:51.942650 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.942656 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:07:51.942689 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 23:07:51.942733 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2690,"thread_start_us":80,"threads_started":1}
I20260924 23:07:51.943647 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.924330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.98.2:36665)
0924 23:07:51.927621 (+ 3291us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.927626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.927635 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:07:51.927675 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.927677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.927695 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.929188 (+ 1493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:51.929272 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.930388 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.940618 (+ 10230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.941432 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.943390 (+ 1958us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.943490 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.943518 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.943564 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3252,"thread_start_us":81,"threads_started":1}
I20260924 23:07:51.944001 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:51.924720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.0.98.1:40863)
0924 23:07:51.927838 (+ 3118us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:51.927843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:51.927852 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:07:51.927884 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:51.927886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:51.927901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:51.928251 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:51.928321 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.931038 (+ 2717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.939610 (+ 8572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:51.940463 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:51.943187 (+ 2724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:51.943278 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:51.943873 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 23:07:51.943930 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3084,"thread_start_us":51,"threads_started":1}
I20260924 23:07:51.944862 11854 tablet_service.cc:1467] Processing CreateTablet for tablet b17566152bc64a55bb1d42288ffda05c (DEFAULT_TABLE table=test-table [id=34509dede017425eb02c818c77080b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:07:51.945247 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17566152bc64a55bb1d42288ffda05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:51.950028 11735 tablet_service.cc:1467] Processing CreateTablet for tablet b17566152bc64a55bb1d42288ffda05c (DEFAULT_TABLE table=test-table [id=34509dede017425eb02c818c77080b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:07:51.950384 11735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b17566152bc64a55bb1d42288ffda05c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:07:51.954205 12093 tablet_bootstrap.cc:492] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Bootstrap starting.
I20260924 23:07:51.954455 12092 tablet_bootstrap.cc:492] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Bootstrap starting.
I20260924 23:07:51.956707 12092 tablet_bootstrap.cc:654] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:51.956943 12094 tablet_bootstrap.cc:492] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: Bootstrap starting.
I20260924 23:07:51.957641 12094 tablet_bootstrap.cc:654] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:51.958451 12093 tablet_bootstrap.cc:654] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Neither blocks nor log segments found. Creating new log.
I20260924 23:07:51.958837 12094 log.cc:824] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:51.959224 12092 log.cc:824] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:51.959972 12093 log.cc:824] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Log is configured to *not* fsync() on all Append() calls
I20260924 23:07:51.961105 12094 tablet_bootstrap.cc:492] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: No bootstrap required, opened a new log
I20260924 23:07:51.961218 12094 ts_tablet_manager.cc:1397] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 23:07:51.964370 12094 raft_consensus.cc:348] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:51.964665 12094 raft_consensus.cc:374] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:51.964756 12094 raft_consensus.cc:729] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 692c77d1cb9047f6b021cf7c09a09470, State: Initialized, Role: FOLLOWER
I20260924 23:07:51.964974 12094 consensus_queue.cc:260] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:51.965332 12094 ts_tablet_manager.cc:1428] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 23:07:51.965499 11801 heartbeater.cc:503] Master 127.0.98.62:42509 was elected leader, sending a full tablet report...
I20260924 23:07:51.966638 12092 tablet_bootstrap.cc:492] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: No bootstrap required, opened a new log
I20260924 23:07:51.966761 12092 ts_tablet_manager.cc:1397] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260924 23:07:51.968921 12092 raft_consensus.cc:348] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:51.969156 12092 raft_consensus.cc:374] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:51.969244 12092 raft_consensus.cc:729] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0ba32419874c12baa77444c841096b, State: Initialized, Role: FOLLOWER
I20260924 23:07:51.969455 12092 consensus_queue.cc:260] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:51.969808 12092 ts_tablet_manager.cc:1428] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:07:51.969993 12073 heartbeater.cc:503] Master 127.0.98.62:42509 was elected leader, sending a full tablet report...
I20260924 23:07:51.971660 12093 tablet_bootstrap.cc:492] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: No bootstrap required, opened a new log
I20260924 23:07:51.971818 12093 ts_tablet_manager.cc:1397] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20260924 23:07:51.974498 12093 raft_consensus.cc:348] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:51.974751 12093 raft_consensus.cc:374] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:07:51.974848 12093 raft_consensus.cc:729] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f464f495fbe64589987021e6ef2fffb8, State: Initialized, Role: FOLLOWER
I20260924 23:07:51.975052 12093 consensus_queue.cc:260] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:51.975414 12093 ts_tablet_manager.cc:1428] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 23:07:51.975574 11938 heartbeater.cc:503] Master 127.0.98.62:42509 was elected leader, sending a full tablet report...
I20260924 23:07:52.061170 12100 raft_consensus.cc:482] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:07:52.061316 12100 raft_consensus.cc:504] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:52.061713 12100 leader_election.cc:288] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0ba32419874c12baa77444c841096b (127.0.98.3:41021), 692c77d1cb9047f6b021cf7c09a09470 (127.0.98.1:40863)
I20260924 23:07:52.066445 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:52.062009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41021 (local address 127.0.98.2:41147)
0924 23:07:52.062201 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:52.062213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:52.062262 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:52.062841 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:52.062842 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:52.062844 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:52.062915 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:52.062917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.063868 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.063869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:52.065380 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:52.065383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.066196 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.066197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:52.066275 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:52.066279 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:07:52.066326 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:07:52.066366 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 23:07:52.067112 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:52.062342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41147 (local address 127.0.98.3:41021)
0924 23:07:52.062386 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:52.062388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:52.062395 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:07:52.062424 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:52.062425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:52.062435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:52.062478 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:52.062996 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.063740 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.065476 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.066081 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.066153 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:52.066164 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:52.067004 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 23:07:52.067046 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:07:52.067384 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17566152bc64a55bb1d42288ffda05c" candidate_uuid: "f464f495fbe64589987021e6ef2fffb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0ba32419874c12baa77444c841096b" is_pre_election: true
I20260924 23:07:52.067533 12020 raft_consensus.cc:2454] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f464f495fbe64589987021e6ef2fffb8 in term 0.
I20260924 23:07:52.067948 11827 leader_election.cc:302] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df0ba32419874c12baa77444c841096b, f464f495fbe64589987021e6ef2fffb8; no voters:
I20260924 23:07:52.068145 12100 raft_consensus.cc:2781] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:07:52.068197 12100 raft_consensus.cc:482] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:07:52.068218 12100 raft_consensus.cc:3037] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:52.069408 12100 raft_consensus.cc:504] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:52.069545 12100 leader_election.cc:288] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [CANDIDATE]: Term 1 election: Requested vote from peers df0ba32419874c12baa77444c841096b (127.0.98.3:41021), 692c77d1cb9047f6b021cf7c09a09470 (127.0.98.1:40863)
I20260924 23:07:52.069834 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17566152bc64a55bb1d42288ffda05c" candidate_uuid: "f464f495fbe64589987021e6ef2fffb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0ba32419874c12baa77444c841096b"
I20260924 23:07:52.069929 12020 raft_consensus.cc:3037] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 0 FOLLOWER]: Advancing to term 1
W20260924 23:07:52.070755 11802 tablet.cc:2369] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:07:52.070778 12020 raft_consensus.cc:2454] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f464f495fbe64589987021e6ef2fffb8 in term 1.
I20260924 23:07:52.071434 11827 leader_election.cc:302] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df0ba32419874c12baa77444c841096b, f464f495fbe64589987021e6ef2fffb8; no voters:
I20260924 23:07:52.071735 12100 raft_consensus.cc:2781] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:07:52.071905 12100 raft_consensus.cc:686] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 1 LEADER]: Becoming Leader. State: Replica: f464f495fbe64589987021e6ef2fffb8, State: Running, Role: LEADER
I20260924 23:07:52.071992 12100 consensus_queue.cc:237] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } }
I20260924 23:07:52.072765 11589 catalog_manager.cc:5495] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f464f495fbe64589987021e6ef2fffb8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f464f495fbe64589987021e6ef2fffb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f464f495fbe64589987021e6ef2fffb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 } health_report { overall_health: UNKNOWN } } }
I20260924 23:07:52.077485 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:52.062608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40863 (local address 127.0.98.2:41309)
0924 23:07:52.066522 (+ 3914us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:52.073850 (+ 7328us) client_negotiation.cc:174] Beginning negotiation
0924 23:07:52.073955 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:07:52.074369 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:07:52.074370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:07:52.074372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:07:52.074415 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:52.074417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.075343 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.075344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:52.076458 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:07:52.076460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.077231 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.077232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:07:52.077303 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:52.077308 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:07:52.077372 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:07:52.077409 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3904,"thread_start_us":57,"threads_started":1}
Sep 24 23:07:52 dist-test-slave-ph9j krb5kdc[11113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:07:52.077692 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:52.062756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41309 (local address 127.0.98.1:40863)
0924 23:07:52.062807 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:52.062808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:52.073923 (+ 11115us) server_negotiation.cc:407] Connection header received
0924 23:07:52.074028 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:52.074029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:52.074042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:52.074083 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:07:52.074493 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.075252 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.076554 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.077135 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.077213 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:52.077224 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:52.077587 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 23:07:52.077632 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:07:52.078060 11755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17566152bc64a55bb1d42288ffda05c" candidate_uuid: "f464f495fbe64589987021e6ef2fffb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692c77d1cb9047f6b021cf7c09a09470" is_pre_election: true
I20260924 23:07:52.078217 11755 raft_consensus.cc:2454] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f464f495fbe64589987021e6ef2fffb8 in term 0.
I20260924 23:07:52.078316 11754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b17566152bc64a55bb1d42288ffda05c" candidate_uuid: "f464f495fbe64589987021e6ef2fffb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "692c77d1cb9047f6b021cf7c09a09470"
I20260924 23:07:52.078361 11754 raft_consensus.cc:3037] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:07:52.079214 11754 raft_consensus.cc:2454] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f464f495fbe64589987021e6ef2fffb8 in term 1.
W20260924 23:07:52.086786 11939 tablet.cc:2369] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:07:52.124974 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:52.116929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.98.2:36665)
0924 23:07:52.117002 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:07:52.117004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:07:52.117013 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:07:52.117062 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:07:52.117063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:07:52.117086 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:07:52.117162 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:07:52.117374 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.118154 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.123650 (+ 5496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:07:52.123995 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:07:52.124077 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:07:52.124312 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:07:52.124464 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:07:52.124540 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 23:07:52.124793 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 23:07:52.124816 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:07:52.130517 11754 raft_consensus.cc:1261] T b17566152bc64a55bb1d42288ffda05c P 692c77d1cb9047f6b021cf7c09a09470 [term 1 FOLLOWER]: Refusing update from remote peer f464f495fbe64589987021e6ef2fffb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:07:52.130987 12102 consensus_queue.cc:1035] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "692c77d1cb9047f6b021cf7c09a09470" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:07:52.133455 12020 raft_consensus.cc:1261] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 1 FOLLOWER]: Refusing update from remote peer f464f495fbe64589987021e6ef2fffb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:07:52.133806 12102 consensus_queue.cc:1035] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0ba32419874c12baa77444c841096b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 23:07:52.135202 12074 tablet.cc:2369] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:07:52.138743 12105 mvcc.cc:205] Tried to move back new op lower bound from 7333033050643046400 to 7333033050407505920. Current Snapshot: MvccSnapshot[applied={T|T < 7333033050643046400}]
I20260924 23:07:52.140132 12108 mvcc.cc:205] Tried to move back new op lower bound from 7333033050643046400 to 7333033050407505920. Current Snapshot: MvccSnapshot[applied={T|T < 7333033050643046400}]
I20260924 23:07:52.140682 12106 mvcc.cc:205] Tried to move back new op lower bound from 7333033050643046400 to 7333033050407505920. Current Snapshot: MvccSnapshot[applied={T|T < 7333033050643046400}]
I20260924 23:07:52.143611 11588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:07:52.143733 11588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:07:52.144661 11588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c135bb5a52be462782808d9d3e8bb589: Sending DeleteTablet for 3 replicas of tablet b17566152bc64a55bb1d42288ffda05c
I20260924 23:07:52.145236 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet b17566152bc64a55bb1d42288ffda05c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:07:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49240
I20260924 23:07:52.145458 11735 tablet_service.cc:1514] Processing DeleteTablet for tablet b17566152bc64a55bb1d42288ffda05c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:07:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40868
I20260924 23:07:52.145823 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet b17566152bc64a55bb1d42288ffda05c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:07:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50784
I20260924 23:07:52.146358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 11670
I20260924 23:07:52.153930 12114 tablet_replica.cc:330] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: stopping tablet replica
I20260924 23:07:52.154079 12114 raft_consensus.cc:2229] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:07:52.154218 12114 raft_consensus.cc:2258] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:07:52.158207 12116 tablet_replica.cc:330] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: stopping tablet replica
I20260924 23:07:52.158362 12116 raft_consensus.cc:2229] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:07:52.158512 12116 raft_consensus.cc:2258] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:07:52.158521 12114 ts_tablet_manager.cc:1905] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:07:52.159040 12116 ts_tablet_manager.cc:1905] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:07:52.160634 12116 ts_tablet_manager.cc:1918] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:07:52.160714 12116 log.cc:1196] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/b17566152bc64a55bb1d42288ffda05c
I20260924 23:07:52.160979 12116 ts_tablet_manager.cc:1939] T b17566152bc64a55bb1d42288ffda05c P f464f495fbe64589987021e6ef2fffb8: Deleting consensus metadata
W20260924 23:07:52.161252 11576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40863 (error 108)
I20260924 23:07:52.161727 11574 catalog_manager.cc:4841] TS f464f495fbe64589987021e6ef2fffb8 (127.0.98.2:36665): tablet b17566152bc64a55bb1d42288ffda05c (table test-table [id=34509dede017425eb02c818c77080b3d]) successfully deleted
I20260924 23:07:52.170403 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:07:52.170215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40863 (local address 127.0.0.1:40870)
0924 23:07:52.170287 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 23:07:52.170337 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 23:07:52.170847 11576 catalog_manager.cc:4584] TS 692c77d1cb9047f6b021cf7c09a09470 (127.0.98.1:40863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b17566152bc64a55bb1d42288ffda05c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40863: connect: Connection refused (error 111)
I20260924 23:07:52.171517 12114 ts_tablet_manager.cc:1918] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:07:52.171615 12114 log.cc:1196] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/b17566152bc64a55bb1d42288ffda05c
I20260924 23:07:52.171962 12114 ts_tablet_manager.cc:1939] T b17566152bc64a55bb1d42288ffda05c P df0ba32419874c12baa77444c841096b: Deleting consensus metadata
I20260924 23:07:52.172489 11575 catalog_manager.cc:4841] TS df0ba32419874c12baa77444c841096b (127.0.98.3:41021): tablet b17566152bc64a55bb1d42288ffda05c (table test-table [id=34509dede017425eb02c818c77080b3d]) successfully deleted
I20260924 23:07:52.174003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 11807
I20260924 23:07:52.179841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 11944
I20260924 23:07:52.185578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 11557
I20260924 23:07:52.191243 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 23:07:52.536027 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 23:07:52.536134 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 23:07:52.881085 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 23:07:52.881 UTC [11143] LOG: received smart shutdown request
2026-09-24 23:07:52.884 UTC [11143] LOG: background worker "logical replication launcher" (PID 11152) exited with exit code 1
2026-09-24 23:07:52.885 UTC [11147] LOG: shutting down
2026-09-24 23:07:53.055 UTC [11143] LOG: database system is shut down
2026-09-24T23:07:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46081 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:07:53 dist-test-slave-ph9j krb5kdc[12126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:07:53 dist-test-slave-ph9j krb5kdc[12126](info): set up 2 sockets
Sep 24 23:07:53 dist-test-slave-ph9j krb5kdc[12126](info): commencing operation
krb5kdc: starting...
W20260924 23:07:54.205788 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:07:54 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:07:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 23:07:54.373044 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKo6W1P/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/postgres -l logfile start
2026-09-24 23:07:56.972 UTC [12156] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 23:07:56.972 UTC [12156] LOG: listening on IPv4 address "127.0.98.20", port 58489
2026-09-24 23:07:56.975 UTC [12156] LOG: listening on Unix socket "/tmp/.s.PGSQL.58489"
2026-09-24 23:07:56.996 UTC [12159] LOG: database system was shut down at 2026-09-24 23:07:55 UTC
2026-09-24 23:07:57.000 UTC [12156] LOG: database system is ready to accept connections
I20260924 23:07:57.957643 392 mini_postgres.cc:96] Postgres bound to 58489
2026-09-24 23:07:57.963 UTC [12167] FATAL: database "slave" does not exist
127.0.98.20:58489 - accepting connections
I20260924 23:07:57.964370 392 mini_ranger.cc:161] Starting Ranger...
I20260924 23:07:57.979140 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 23:07:58.466096 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 23:07:58.466168 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-admin
2026-09-24 23:07:58,752 [I] DB FLAVOR :POSTGRES
2026-09-24 23:07:58,753 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:07:58,753 [I] Checking connection..
2026-09-24 23:07:58,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 23:07:58,991 [I] Checking connection passed.
2026-09-24 23:07:58,991 [I] --------- Verifying version history table ---------
2026-09-24 23:07:58,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:07:59,260 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:07:59,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:07:59,530 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 23:07:59,531 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 23:07:59,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 23:07:59.793 UTC [12231] WARNING: there is no transaction in progress
2026-09-24 23:07:59,799 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 23:07:59,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 23:08:00,038 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 23:08:00,038 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:08:00,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 23:08:00,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 23:08:00,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ph9j.c.gcp-upstream.internal','N') ;"
2026-09-24 23:08:00,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:08:01,053 [I] Table x_portal_user does not exist in database ranger
2026-09-24 23:08:01,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:08:01,317 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 23:08:01,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:08:01,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:08:01,801 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 23:08:01,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 23:08:02.826 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.842 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.859 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.875 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.892 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.911 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.967 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.974 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:02.990 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.001 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.012 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.026 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.036 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.042 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.049 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03.056 UTC [12339] WARNING: there is no transaction in progress
2026-09-24 23:08:03,745 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 23:08:03,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 23:08:03,990 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 23:08:03,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 23:08:04,233 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 23:08:04,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 23:08:04,471 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 23:08:04,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 23:08:04,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 23:08:04,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ph9j.c.gcp-upstream.internal';"
2026-09-24 23:08:05,224 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 23:08:05,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 23:08:05,475 [I] DB_PATCHES have already been applied
I20260924 23:08:05.480630 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKo6W1P/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 23:08:05.480702 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 23:08:05.483763 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39773
Sep 24, 2026 11:08:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:08:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:08:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 11:08:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 11:08:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 11:08:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 11:08:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 23:08:06.611979 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.648s user 0.000s sys 0.008s
Sep 24, 2026 11:08:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 11:08:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 11:08:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 11:08:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 11:08:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 23:08:27.932842 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 23:08:28 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291308, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 23:08:28.023217 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 23:08:28.032722 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 23:08:28.948395 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 23:08:28.948522 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms
I20260924 23:08:28.948547 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58489
2026-09-24 23:08:29,108 [I] DB FLAVOR :POSTGRES
2026-09-24 23:08:29,109 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 23:08:29,109 [I] Checking connection
2026-09-24 23:08:29,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 23:08:29,354 [I] connection success
2026-09-24 23:08:29,354 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 23:08:29,354 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 23:08:29,354 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 23:08:29,616 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 23:08:29,616 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 23:08:29,616 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 23:08:29,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58489/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 23:08:29,887 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 23:08:30.384011 392 mini_ranger_kms.cc:310] Created kms service
I20260924 23:08:30.562325 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 23:08:30.607640 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 23:08:30 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291308, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:31.036072 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 23:08:31.036270 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKo6W1P/build/release/bin/postgresql.jar:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKo6W1P/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 23:08:31.036291 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 23:08:31.039539 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12519
I20260924 23:08:31.039611 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:50007
Sep 24, 2026 11:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 11:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 11:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 11:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 11:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 11:08:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 23:08:33.304553 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.281s user 0.004s sys 0.138s
I20260924 23:08:33.304672 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 23:08:33.304728 392 mini_ranger_kms.cc:401] 127.0.98.20:50007/kms/v1/keys
Sep 24 23:08:33 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 23:08:33 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 23:08:34 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 23:08:36 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:08:36.499135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33145
--ranger_config_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:36.582536 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:36.582732 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:36.582753 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:36.583804 12570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:36.583837 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:36.583851 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:36.583863 12570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:36.583875 12570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:36.585333 12570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:50007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33145
--ranger_config_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:36.585659 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:36.585896 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:36.590937 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:36.590972 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:36.591044 12570 server_base.cc:1034] running on GCE node
W20260924 23:08:36.591198 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:36.591503 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:36.591792 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:36.592983 12570 hybrid_clock.cc:648] HybridClock initialized: now 1790291316592972 us; error 51 us; skew 500 ppm
Sep 24 23:08:36 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:36.600873 12570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:36.601375 12570 webserver.cc:460] Webserver started at http://127.0.98.62:35453/ using document root <none> and password file <none>
I20260924 23:08:36.601567 12570 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:36.601603 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:36.601693 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:08:36 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:36.653165 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "524f82d372244e3199d95014f4a14316"
format_stamp: "Formatted at 2026-09-24 23:08:36 on dist-test-slave-ph9j"
server_key: "96b84d8a14a90c162590bf6044468dc5"
server_key_iv: "bc6521eac46e52d16cee64f1e26bebbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:36.653663 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "524f82d372244e3199d95014f4a14316"
format_stamp: "Formatted at 2026-09-24 23:08:36 on dist-test-slave-ph9j"
server_key: "96b84d8a14a90c162590bf6044468dc5"
server_key_iv: "bc6521eac46e52d16cee64f1e26bebbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:36.655602 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 23:08:36.670372 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:36.670562 12570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:36.670655 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "524f82d372244e3199d95014f4a14316"
format_stamp: "Formatted at 2026-09-24 23:08:36 on dist-test-slave-ph9j"
server_key: "96b84d8a14a90c162590bf6044468dc5"
server_key_iv: "bc6521eac46e52d16cee64f1e26bebbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:36.670749 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:36.691743 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:36.693993 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:36.694113 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:36.698294 12570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33145
I20260924 23:08:36.698340 12637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33145 every 8 connection(s)
I20260924 23:08:36.698686 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:36.699411 12638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:36.701772 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Bootstrap starting.
I20260924 23:08:36.702392 12638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:36.702682 12638 log.cc:824] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:36.703341 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: No bootstrap required, opened a new log
I20260924 23:08:36.705044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 12570
I20260924 23:08:36.705173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:08:36.705377 392 external_mini_cluster.cc:1428] Setting key bc9267a03e83263c0fba954a6e6ca7ef
I20260924 23:08:36.705643 12638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:36.705812 12638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:36.705875 12638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f82d372244e3199d95014f4a14316, State: Initialized, Role: FOLLOWER
I20260924 23:08:36.706079 12638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:36.706175 12638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:08:36.706211 12638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:08:36.706256 12638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:36.706902 12638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:36.706997 12638 leader_election.cc:302] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524f82d372244e3199d95014f4a14316; no voters:
I20260924 23:08:36.707218 12638 leader_election.cc:288] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:08:36.707240 12643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:36.707350 12643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 LEADER]: Becoming Leader. State: Replica: 524f82d372244e3199d95014f4a14316, State: Running, Role: LEADER
I20260924 23:08:36.707438 12643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:36.707486 12638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:36.707774 12643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "524f82d372244e3199d95014f4a14316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } } }
I20260924 23:08:36.707846 12643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:36.707966 12643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524f82d372244e3199d95014f4a14316. Latest consensus state: current_term: 1 leader_uuid: "524f82d372244e3199d95014f4a14316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } } }
I20260924 23:08:36.708045 12643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:36.708338 12648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:36.708916 12648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:36.709183 12638 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 23:08:36.713366 12648 catalog_manager.cc:1347] Generated new cluster ID: 77dede29f99242b09b91a5934dbcbb16
I20260924 23:08:36.713536 12648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:08:36 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:36.721344 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:36.705913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.0.98.62:33145)
0924 23:08:36.706129 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:36.706139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:36.706175 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:08:36.706296 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:36.706303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:36.706363 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:36.706537 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:36.707174 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:36.708679 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:36.709762 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:36.712485 (+ 2723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:36.715930 (+ 3445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:36.716044 (+ 114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:36.716047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:36.716048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:36.716058 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:36.719061 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:36.719505 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:36.719509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:36.719510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:36.719638 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:36.719892 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:36.719893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:36.719893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:36.720247 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:36.720881 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:36.720965 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:08:36.721092 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260924 23:08:36.729754 12648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:08:36.730485 12648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:08:36.740360 12648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Generated new TSK 0
I20260924 23:08:36.740664 12648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:08:37.448915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33145
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:37.554234 12683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:37.554446 12683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:37.554477 12683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:37.555718 12683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:37.555761 12683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:37.555788 12683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:08:37.557283 12683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:50007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:37.557534 12683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:37.557789 12683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:37.563867 12691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:37.563951 12689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:37.564043 12683 server_base.cc:1034] running on GCE node
W20260924 23:08:37.563851 12688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:37.564338 12683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:37.564654 12683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:37.565842 12683 hybrid_clock.cc:648] HybridClock initialized: now 1790291317565825 us; error 34 us; skew 500 ppm
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:37.569002 12683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:08:37.569531 12683 webserver.cc:460] Webserver started at http://127.0.98.1:46389/ using document root <none> and password file <none>
I20260924 23:08:37.569799 12683 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:37.569866 12683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:37.570041 12683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:37.585551 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "2c26d8f27a4c42f29ff9110fd11ec01b"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "b92927df1f01d7508bdf9d42f1484923"
server_key_iv: "c199bdbfce4024d0cf3f14a41c575378"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:37.586161 12683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2c26d8f27a4c42f29ff9110fd11ec01b"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "b92927df1f01d7508bdf9d42f1484923"
server_key_iv: "c199bdbfce4024d0cf3f14a41c575378"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:37.588066 12683 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 23:08:37.607257 12699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:37.607963 12683 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 23:08:37.608068 12683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "2c26d8f27a4c42f29ff9110fd11ec01b"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "b92927df1f01d7508bdf9d42f1484923"
server_key_iv: "c199bdbfce4024d0cf3f14a41c575378"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:37.608178 12683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:37.660822 12683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:37.663426 12683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:37.663661 12683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:37.663924 12683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:37.664323 12683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:37.664384 12683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:37.664450 12683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:37.664494 12683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:37.671658 12683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35477
I20260924 23:08:37.672045 12683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:08:37.672540 12813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35477 every 8 connection(s)
I20260924 23:08:37.676909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 12683
I20260924 23:08:37.677016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:08:37.677187 392 external_mini_cluster.cc:1428] Setting key 93030df5352bfd7aa1f5b768db626309
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:37.685722 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:37.672805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.98.1:33829)
0924 23:08:37.673160 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:37.673209 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:37.673741 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:37.674093 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:37.674096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:37.674313 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:37.674527 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:37.674532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.675466 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.675468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:37.677431 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:37.677435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.678015 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.678016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:37.678152 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:37.678753 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:37.678764 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:37.681937 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:37.683972 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:37.683979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:37.683982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:37.684282 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:37.684529 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:37.684530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:37.684530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:37.684684 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:37.685001 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:37.685007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:37.685187 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 23:08:37.685300 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:08:37.685460 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":80,"threads_started":1}
I20260924 23:08:37.686291 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:37.673118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33829 (local address 127.0.98.62:33145)
0924 23:08:37.673247 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:37.673250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:37.673288 (+ 38us) server_negotiation.cc:407] Connection header received
0924 23:08:37.673869 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:37.673871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:37.673913 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:37.673976 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:37.674659 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.675331 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.677530 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.677827 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.677924 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:37.682049 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:37.682050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:37.682051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:37.682059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:37.683882 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:37.684383 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:37.684384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:37.684385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:37.684449 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:37.684759 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:37.684760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:37.684760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:37.684898 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:37.684963 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:37.686113 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0924 23:08:37.686197 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260924 23:08:37.687069 12814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33145
I20260924 23:08:37.687162 12814 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:37.687345 12814 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:37.687913 12602 ts_manager.cc:194] Registered new tserver with Master: 2c26d8f27a4c42f29ff9110fd11ec01b (127.0.98.1:35477)
I20260924 23:08:37.688810 12602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33829
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:08:37.752293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33145
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 23:08:37.885531 12822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:37.885803 12822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:37.885834 12822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:37.887562 12822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:37.887619 12822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:37.887656 12822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:08:37.889755 12822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:50007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:37.889999 12822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:37.890275 12822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:37.896422 12827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:37.896464 12830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:37.896693 12828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:37.897447 12822 server_base.cc:1034] running on GCE node
I20260924 23:08:37.897652 12822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:37.897912 12822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:37.899080 12822 hybrid_clock.cc:648] HybridClock initialized: now 1790291317899052 us; error 40 us; skew 500 ppm
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:37.902357 12822 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:08:37.902983 12822 webserver.cc:460] Webserver started at http://127.0.98.2:37099/ using document root <none> and password file <none>
I20260924 23:08:37.903268 12822 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:37.903393 12822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:37.903592 12822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:37.923342 12822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc54a6e4d70d460888f321f47d94657a"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "7ab8e6969311a42d7c8174043cfe7e20"
server_key_iv: "4438e3d9e9be04ad32372c29df3b3de2"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:37.923887 12822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc54a6e4d70d460888f321f47d94657a"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "7ab8e6969311a42d7c8174043cfe7e20"
server_key_iv: "4438e3d9e9be04ad32372c29df3b3de2"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:37.925868 12822 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 23:08:37.943867 12838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:37.944118 12822 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260924 23:08:37.944207 12822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "bc54a6e4d70d460888f321f47d94657a"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "7ab8e6969311a42d7c8174043cfe7e20"
server_key_iv: "4438e3d9e9be04ad32372c29df3b3de2"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:37.944298 12822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:37.960389 12822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:37.962838 12822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:37.962997 12822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:37.963217 12822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:37.963632 12822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:37.963686 12822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:37.963717 12822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:37.963775 12822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:37.970453 12822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44487
I20260924 23:08:37.970953 12822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:08:37.971707 12951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44487 every 8 connection(s)
I20260924 23:08:37.973899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 12822
I20260924 23:08:37.974017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:08:37.974203 392 external_mini_cluster.cc:1428] Setting key 5092ccbcb93b8e0756ab5e2e16d4540a
Sep 24 23:08:37 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:37.985775 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:37.972630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46567 (local address 127.0.98.62:33145)
0924 23:08:37.972673 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:37.972675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:37.973140 (+ 465us) server_negotiation.cc:407] Connection header received
0924 23:08:37.973964 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:37.973966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:37.973982 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:37.974018 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:37.977049 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.977804 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.979427 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.979763 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.979801 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:37.982309 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:37.982310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:37.982311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:37.982320 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:37.984102 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:37.984682 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:37.984683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:37.984684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:37.984748 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:37.985036 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:37.985037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:37.985037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:37.985187 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:37.985243 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:37.985581 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 23:08:37.985671 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:37.986039 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:37.972346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.98.2:46567)
0924 23:08:37.973010 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:37.973052 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:37.973750 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:37.974228 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:37.974232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:37.974538 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:37.974855 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:37.974862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.977956 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.977957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:37.979311 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:37.979315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:37.979885 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:37.979886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:37.979983 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:37.980556 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:37.980565 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:37.982192 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:37.984192 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:37.984198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:37.984201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:37.984588 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:37.984818 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:37.984819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:37.984819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:37.984956 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:37.985276 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:37.985282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:37.985426 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 23:08:37.985549 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 23:08:37.985750 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":91,"threads_started":1}
I20260924 23:08:37.989106 12952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33145
I20260924 23:08:37.989248 12952 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:37.990016 12952 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:37.990414 12602 ts_manager.cc:194] Registered new tserver with Master: bc54a6e4d70d460888f321f47d94657a (127.0.98.2:44487)
I20260924 23:08:37.992152 12602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46567
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:08:38.026654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33145
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:38.152007 12957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:38.152248 12957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:38.152287 12957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:38.153468 12957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:38.153538 12957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:38.153580 12957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:08:38.155109 12957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:50007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:38.155377 12957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:38.155615 12957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:38.160704 12965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.160780 12963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.160895 12962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:38.161204 12957 server_base.cc:1034] running on GCE node
I20260924 23:08:38.161405 12957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:38.161679 12957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:38.165221 12957 hybrid_clock.cc:648] HybridClock initialized: now 1790291318165200 us; error 34 us; skew 500 ppm
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:38.168565 12957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:08:38.169140 12957 webserver.cc:460] Webserver started at http://127.0.98.3:41575/ using document root <none> and password file <none>
I20260924 23:08:38.169481 12957 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:38.169596 12957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:38.169771 12957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:38.187637 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "f396619d26ef4129823e58a468a91aca"
format_stamp: "Formatted at 2026-09-24 23:08:38 on dist-test-slave-ph9j"
server_key: "feeb9b4bbb3253420deaea90ac6cc19c"
server_key_iv: "6ab69067fd0504ca90ace00812625783"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:38.188254 12957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f396619d26ef4129823e58a468a91aca"
format_stamp: "Formatted at 2026-09-24 23:08:38 on dist-test-slave-ph9j"
server_key: "feeb9b4bbb3253420deaea90ac6cc19c"
server_key_iv: "6ab69067fd0504ca90ace00812625783"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:38.201376 12957 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.000s sys 0.002s
I20260924 23:08:38.220158 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.220422 12957 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 23:08:38.220548 12957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "f396619d26ef4129823e58a468a91aca"
format_stamp: "Formatted at 2026-09-24 23:08:38 on dist-test-slave-ph9j"
server_key: "feeb9b4bbb3253420deaea90ac6cc19c"
server_key_iv: "6ab69067fd0504ca90ace00812625783"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:38.220649 12957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:38.231612 12957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:38.234092 12957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:38.234301 12957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:38.234614 12957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:38.235047 12957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:38.235136 12957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.235215 12957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:38.235276 12957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.241607 12957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42393
I20260924 23:08:38.242108 12957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:08:38.245016 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42393 every 8 connection(s)
I20260924 23:08:38.249575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 12957
I20260924 23:08:38.249681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:08:38.249855 392 external_mini_cluster.cc:1428] Setting key d4c1b1619118796827c0c0ba8646ebb6
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:38.256609 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:38.245993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59439 (local address 127.0.98.62:33145)
0924 23:08:38.246046 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:38.246049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:38.246274 (+ 225us) server_negotiation.cc:407] Connection header received
0924 23:08:38.246986 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:38.246988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:38.247001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:38.247033 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:38.247888 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.248663 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.250369 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.250673 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.250734 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:38.253339 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:38.253341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:38.253341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:38.253349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:38.255034 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:38.255501 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:38.255502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:38.255503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:38.255554 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:38.255851 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:38.255852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:38.255852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:38.255987 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:38.256031 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:38.256409 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 23:08:38.256508 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:08:38.257330 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:38.245695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.98.3:59439)
0924 23:08:38.246144 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:38.246188 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:38.246857 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:38.247224 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:38.247228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:38.247484 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:38.247759 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:38.247765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.248830 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.248831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:38.250264 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:38.250266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.250840 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.250841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:38.250986 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:38.251577 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:38.251587 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:38.253227 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:38.255125 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:38.255131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:38.255133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:38.255407 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:38.255624 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:38.255625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:38.255625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:38.255760 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:38.256080 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:38.256086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:38.256256 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 23:08:38.256922 (+ 666us) client_negotiation.cc:240] Negotiation successful
0924 23:08:38.257104 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":73,"threads_started":1}
I20260924 23:08:38.258037 13086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33145
I20260924 23:08:38.258128 13086 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.258307 13086 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:38.258741 12602 ts_manager.cc:194] Registered new tserver with Master: f396619d26ef4129823e58a468a91aca (127.0.98.3:42393)
I20260924 23:08:38.259199 12602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59439
I20260924 23:08:38.262234 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:08:38.262885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 12683
I20260924 23:08:38.269052 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 12822
I20260924 23:08:38.276176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 12957
I20260924 23:08:38.281152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 12570
I20260924 23:08:38.286971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33145
--webserver_interface=127.0.98.62
--webserver_port=35453
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33145
--ranger_config_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:38.369307 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:38.369505 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:38.369526 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:38.370651 13089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:38.370688 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:38.370702 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:38.370715 13089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:38.370728 13089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:38.372274 13089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33145
--ranger_config_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:38.372504 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:38.372738 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:38.377703 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.377818 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:38.377925 13089 server_base.cc:1034] running on GCE node
W20260924 23:08:38.377703 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:38.378304 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:38.378631 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:38.379807 13089 hybrid_clock.cc:648] HybridClock initialized: now 1790291318379790 us; error 32 us; skew 500 ppm
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:38.382865 13089 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:38.383338 13089 webserver.cc:460] Webserver started at http://127.0.98.62:35453/ using document root <none> and password file <none>
I20260924 23:08:38.383651 13089 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:38.383742 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:38.447096 13089 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:38.447937 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.448143 13089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:38.448259 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "524f82d372244e3199d95014f4a14316"
format_stamp: "Formatted at 2026-09-24 23:08:36 on dist-test-slave-ph9j"
server_key: "96b84d8a14a90c162590bf6044468dc5"
server_key_iv: "bc6521eac46e52d16cee64f1e26bebbb"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:38.448633 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:38.464815 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:38.466929 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:38.467078 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:38.472007 13089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33145
I20260924 23:08:38.472090 13157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33145 every 8 connection(s)
I20260924 23:08:38.472463 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:38.473151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13089
I20260924 23:08:38.473469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35477
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33145
--webserver_port=46389
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:38.474280 13158 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:08:38.475248 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Bootstrap starting.
I20260924 23:08:38.486296 13158 log.cc:824] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:38.488734 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:08:38.489179 13158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Bootstrap complete.
I20260924 23:08:38.492534 13158 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:38.492928 13158 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524f82d372244e3199d95014f4a14316, State: Initialized, Role: FOLLOWER
I20260924 23:08:38.493198 13158 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:38.493341 13158 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:08:38.493474 13158 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:08:38.493582 13158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:08:38.494982 13158 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:38.495152 13158 leader_election.cc:302] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524f82d372244e3199d95014f4a14316; no voters:
I20260924 23:08:38.495370 13158 leader_election.cc:288] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:08:38.495455 13162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:08:38.495669 13158 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:38.495675 13162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [term 2 LEADER]: Becoming Leader. State: Replica: 524f82d372244e3199d95014f4a14316, State: Running, Role: LEADER
I20260924 23:08:38.495869 13162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } }
I20260924 23:08:38.496104 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524f82d372244e3199d95014f4a14316. Latest consensus state: current_term: 2 leader_uuid: "524f82d372244e3199d95014f4a14316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } } }
I20260924 23:08:38.496161 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:38.496395 13164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "524f82d372244e3199d95014f4a14316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524f82d372244e3199d95014f4a14316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33145 } } }
I20260924 23:08:38.496505 13164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:38.496641 13167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:38.497157 13167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:38.497345 13167 catalog_manager.cc:1259] Loaded cluster ID: 77dede29f99242b09b91a5934dbcbb16
I20260924 23:08:38.497418 13167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:08:38.497493 13158 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 23:08:38.498770 13167 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:08:38.499010 13167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Loaded TSK: 0
I20260924 23:08:38.499301 13167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:08:38.591455 13160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:38.592046 13160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:38.592149 13160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:38.594059 13160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:38.594151 13160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:38.594233 13160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:08:38.596344 13160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35477
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46389
--tserver_master_addrs=127.0.98.62:33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:38.596866 13160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:38.597196 13160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:38.603940 13195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.604228 13198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.605173 13196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:38.605298 13160 server_base.cc:1034] running on GCE node
I20260924 23:08:38.605589 13160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:38.605859 13160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:38.607080 13160 hybrid_clock.cc:648] HybridClock initialized: now 1790291318607057 us; error 35 us; skew 500 ppm
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:38.610263 13160 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:08:38.610743 13160 webserver.cc:460] Webserver started at http://127.0.98.1:46389/ using document root <none> and password file <none>
I20260924 23:08:38.610993 13160 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:38.611060 13160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:38.633524 13160 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 23:08:38.634390 13206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.634588 13160 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:38.634666 13160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "2c26d8f27a4c42f29ff9110fd11ec01b"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "b92927df1f01d7508bdf9d42f1484923"
server_key_iv: "c199bdbfce4024d0cf3f14a41c575378"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:38.635092 13160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:38.653102 13160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:38.656038 13160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:38.656265 13160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:38.656535 13160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:38.656962 13160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:38.657074 13160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.657186 13160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:38.657270 13160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.664928 13160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35477
I20260924 23:08:38.665517 13160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:08:38.668385 13319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35477 every 8 connection(s)
I20260924 23:08:38.673501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13160
I20260924 23:08:38.673840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44487
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33145
--webserver_port=37099
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:38.687057 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:38.669303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42433 (local address 127.0.98.62:33145)
0924 23:08:38.669551 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:38.669562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:38.670200 (+ 638us) server_negotiation.cc:407] Connection header received
0924 23:08:38.671144 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:38.671153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:38.671223 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:38.671338 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:38.672331 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.676812 (+ 4481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.678527 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.679283 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.679392 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:38.682426 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:38.682431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:38.682431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:38.682443 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:38.684791 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:38.685329 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:38.685332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:38.685333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:38.685443 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:38.685752 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:38.685753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:38.685753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:38.686113 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:38.686194 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:38.686638 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 23:08:38.686774 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
I20260924 23:08:38.688102 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:38.669198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.98.1:42433)
0924 23:08:38.670025 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:38.670075 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:38.670821 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:38.671598 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:38.671602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:38.671875 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:38.672182 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:38.672189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.676982 (+ 4793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.676983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:38.678405 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:38.678409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.679553 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.679554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:38.679703 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:38.680377 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:38.680389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:38.682271 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:38.684890 (+ 2619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:38.684896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:38.684899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:38.685221 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:38.685527 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:38.685528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:38.685528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:38.685664 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:38.686235 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:38.686241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:38.686413 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 23:08:38.687682 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0924 23:08:38.687858 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":79,"threads_started":1}
I20260924 23:08:38.689209 13320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33145
I20260924 23:08:38.689307 13320 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.689495 13320 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
W20260924 23:08:38.690454 13320 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33145 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 23:08:38.690529 13320 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.690598 13320 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:38.691012 13320 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.691092 13320 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
W20260924 23:08:38.691452 13320 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:08:38.822670 13324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:38.823009 13324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:38.823053 13324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:38.824893 13324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:38.824965 13324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:38.825004 13324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:08:38.827314 13324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37099
--tserver_master_addrs=127.0.98.62:33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:38.827569 13324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:38.827914 13324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:38.834334 13332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.834383 13329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:38.834511 13330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:38.835073 13324 server_base.cc:1034] running on GCE node
I20260924 23:08:38.835300 13324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:38.835566 13324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:38.836745 13324 hybrid_clock.cc:648] HybridClock initialized: now 1790291318836712 us; error 50 us; skew 500 ppm
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:38.840114 13324 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:08:38.840621 13324 webserver.cc:460] Webserver started at http://127.0.98.2:37099/ using document root <none> and password file <none>
I20260924 23:08:38.840895 13324 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:38.840973 13324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:38.861088 13324 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 23:08:38.864583 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.864863 13324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 23:08:38.865003 13324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "bc54a6e4d70d460888f321f47d94657a"
format_stamp: "Formatted at 2026-09-24 23:08:37 on dist-test-slave-ph9j"
server_key: "7ab8e6969311a42d7c8174043cfe7e20"
server_key_iv: "4438e3d9e9be04ad32372c29df3b3de2"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:38.865537 13324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:38.878940 13324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:38.880916 13324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:38.881088 13324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:38.881340 13324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:38.881701 13324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:38.881755 13324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.881798 13324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:38.881824 13324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:38.889087 13324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44487
I20260924 23:08:38.889600 13324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:08:38.890169 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44487 every 8 connection(s)
I20260924 23:08:38.894588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13324
I20260924 23:08:38.894869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:50007/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42393
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33145
--webserver_port=41575
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:08:38 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:38.901424 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:38.890399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.98.2:42731)
0924 23:08:38.890783 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:38.890829 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:38.891466 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:38.891768 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:38.891772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:38.892034 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:38.892342 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:38.892350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.893374 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.893375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:38.894746 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:38.894749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.895306 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.895308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:38.895455 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:38.896118 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:38.896129 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:38.897676 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:38.899704 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:38.899711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:38.899713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:38.900004 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:38.900298 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:38.900299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:38.900299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:38.900428 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:38.900739 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:38.900745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:38.900915 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 23:08:38.901028 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:08:38.901196 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":86,"threads_started":1}
I20260924 23:08:38.901934 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:38.890778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42731 (local address 127.0.98.62:33145)
0924 23:08:38.890828 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:38.890830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:38.890930 (+ 100us) server_negotiation.cc:407] Connection header received
0924 23:08:38.891599 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:38.891600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:38.891616 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:38.891700 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:38.892464 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.893259 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.894857 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:38.895144 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:38.895195 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:38.897805 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:38.897807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:38.897807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:38.897815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:38.899578 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:38.900106 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:38.900108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:38.900108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:38.900170 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:38.900512 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:38.900513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:38.900513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:38.900649 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:38.900711 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:38.901730 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0924 23:08:38.901820 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 23:08:38.902779 13454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33145
I20260924 23:08:38.902882 13454 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.903069 13454 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
W20260924 23:08:38.903721 13454 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33145 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 23:08:38.903788 13454 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.903853 13454 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:38.904177 13454 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:38.904248 13454 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
W20260924 23:08:38.904551 13454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:08:39.020010 13457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:39.020294 13457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:39.020337 13457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:39.022099 13457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:39.022213 13457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:39.022266 13457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:08:39.024506 13457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:50007/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41575
--tserver_master_addrs=127.0.98.62:33145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:39.024783 13457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:39.025099 13457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:39.031801 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:39.031782 13463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:39.031773 13465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:39.032577 13457 server_base.cc:1034] running on GCE node
I20260924 23:08:39.032788 13457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:39.033061 13457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:39.034225 13457 hybrid_clock.cc:648] HybridClock initialized: now 1790291319034204 us; error 37 us; skew 500 ppm
Sep 24 23:08:39 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:39.037559 13457 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:08:39.038110 13457 webserver.cc:460] Webserver started at http://127.0.98.3:41575/ using document root <none> and password file <none>
I20260924 23:08:39.038379 13457 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:39.038457 13457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 23:08:39 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:39.056565 13457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:39.057408 13474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:39.057564 13457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:39.057637 13457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "f396619d26ef4129823e58a468a91aca"
format_stamp: "Formatted at 2026-09-24 23:08:38 on dist-test-slave-ph9j"
server_key: "feeb9b4bbb3253420deaea90ac6cc19c"
server_key_iv: "6ab69067fd0504ca90ace00812625783"
server_key_version: "kuduclusterkey@0"
I20260924 23:08:39.058177 13457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:39.085188 13457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:39.087466 13457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:39.087642 13457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:39.087889 13457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:39.088292 13457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:39.088348 13457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:39.088413 13457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:39.088445 13457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:39.095296 13457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42393
I20260924 23:08:39.095710 13587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42393 every 8 connection(s)
I20260924 23:08:39.095857 13457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 23:08:39 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:39.103863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13457
I20260924 23:08:39.107494 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:39.096737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35567 (local address 127.0.98.62:33145)
0924 23:08:39.096785 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:39.096787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:39.097181 (+ 394us) server_negotiation.cc:407] Connection header received
0924 23:08:39.097851 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:39.097853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:39.097866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:39.097972 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:39.098752 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:39.099530 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:39.101076 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:39.101362 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:39.101445 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:39.104095 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:39.104096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:39.104097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:39.104105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:39.105850 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:39.106387 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:39.106389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:39.106389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:39.106452 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:39.106727 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:39.106728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:39.106729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:39.106854 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:39.106896 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:39.107303 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 23:08:39.107391 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:08:39.108031 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:39.096742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33145 (local address 127.0.98.3:35567)
0924 23:08:39.096994 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:39.097079 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:39.097724 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:39.098111 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:39.098116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:39.098359 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:39.098636 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:39.098642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:39.099644 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:39.099645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:39.100969 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:39.100972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:39.101484 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:39.101485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:39.101654 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:39.102317 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:39.102327 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:39.103980 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:39.105996 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:39.106003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:39.106006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:39.106294 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:39.106524 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:39.106525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:39.106525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:39.106654 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:39.106956 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:39.106962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:39.107140 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 23:08:39.107626 (+ 486us) client_negotiation.cc:240] Negotiation successful
0924 23:08:39.107789 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260924 23:08:39.108613 13588 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33145
I20260924 23:08:39.108731 13588 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:39.108914 13588 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
W20260924 23:08:39.109529 13588 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33145 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 23:08:39.109606 13588 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:39.109670 13588 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:39.110026 13588 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:39.110093 13588 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
W20260924 23:08:39.110370 13588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:08:39.114845 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:39.104442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.0.98.62:33145)
0924 23:08:39.104940 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:39.104942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:39.104950 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:08:39.104969 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:39.104970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:39.105244 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:39.105319 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:39.106210 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:39.106981 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:39.109349 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:39.110262 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:39.110321 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:39.111862 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:39.111864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:39.111864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:39.111871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:39.113617 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:39.114045 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:39.114046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:39.114046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:39.114110 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:39.114316 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:39.114317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:39.114317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:39.114455 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:39.114626 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:39.114662 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 23:08:39.114743 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":59,"threads_started":1}
Sep 24 23:08:39 dist-test-slave-ph9j krb5kdc[12126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790291319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:39.691645 13320 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:39.691804 13320 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:39.692209 13122 ts_manager.cc:194] Registered new tserver with Master: 2c26d8f27a4c42f29ff9110fd11ec01b (127.0.98.1:35477)
I20260924 23:08:39.693225 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42433
I20260924 23:08:39.904726 13454 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:39.904908 13454 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:39.905267 13122 ts_manager.cc:194] Registered new tserver with Master: bc54a6e4d70d460888f321f47d94657a (127.0.98.2:44487)
I20260924 23:08:39.905710 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42731
Sep 24 23:08:40 dist-test-slave-ph9j krb5kdc[12126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 23:08:40.110545 13588 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:40.110693 13588 heartbeater.cc:511] Master 127.0.98.62:33145 requested a full tablet report, sending...
I20260924 23:08:40.111029 13122 ts_manager.cc:194] Registered new tserver with Master: f396619d26ef4129823e58a468a91aca (127.0.98.3:42393)
I20260924 23:08:40.111526 13122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35567
I20260924 23:08:40.119053 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:08:40.127944 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.120041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.98.62:33145)
0924 23:08:40.120330 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.120333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.120490 (+ 157us) server_negotiation.cc:407] Connection header received
0924 23:08:40.120514 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.120515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.120527 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.120578 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:40.121235 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.122028 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.123009 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.123306 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.123358 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.124855 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:40.124857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:40.124857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:40.124865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:40.126591 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:40.126944 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:40.126945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:40.126945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:40.126999 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:40.127165 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:40.127166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:40.127166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:40.127282 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:40.127361 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.127764 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.127843 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":73,"threads_started":1}
I20260924 23:08:40.130340 13122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:08:40.132514 13122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:08:40.147612 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.140666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44487 (local address 127.0.0.1:38696)
0924 23:08:40.142616 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.142624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:40.142694 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:40.143313 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:40.143316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:40.143318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:40.143395 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.143397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.144600 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.144601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.146263 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.146265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.147299 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.147300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.147377 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.147393 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 23:08:40.147458 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:08:40.147516 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1935,"thread_start_us":76,"threads_started":1}
I20260924 23:08:40.148370 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.140962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.0.98.2:44487)
0924 23:08:40.141310 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.141315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.142669 (+ 1354us) server_negotiation.cc:407] Connection header received
0924 23:08:40.142775 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.142778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.142795 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.143176 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:40.143475 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.144489 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.146353 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.147115 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.147167 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.147238 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.147941 (+ 703us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.148284 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":74,"threads_started":1}
I20260924 23:08:40.149394 13388 tablet_service.cc:1467] Processing CreateTablet for tablet 9f72197fde394333ab74b44e678b4ebd (DEFAULT_TABLE table=test-table [id=91a2292bdecc4118b25f84b91cc64ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:40.149794 13388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f72197fde394333ab74b44e678b4ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:40.153228 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.140509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35477 (local address 127.0.0.1:47178)
0924 23:08:40.142488 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.142496 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:40.142544 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:40.149185 (+ 6641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:40.149186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:40.149189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:40.149228 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.149229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.150459 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.150461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.151953 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.151955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.152968 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.152969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.153061 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.153067 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:08:40.153103 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 23:08:40.153144 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1964,"thread_start_us":59,"threads_started":1}
I20260924 23:08:40.153680 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.140607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.0.98.1:35477)
0924 23:08:40.149014 (+ 8407us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.149019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.149031 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:08:40.149065 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.149067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.149083 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.149298 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:40.149364 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.150358 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.152064 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.152854 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.153395 (+ 541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.153517 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.153546 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.153596 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8357,"thread_start_us":54,"threads_started":1}
I20260924 23:08:40.154466 13254 tablet_service.cc:1467] Processing CreateTablet for tablet 9f72197fde394333ab74b44e678b4ebd (DEFAULT_TABLE table=test-table [id=91a2292bdecc4118b25f84b91cc64ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:40.154853 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f72197fde394333ab74b44e678b4ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:40.159328 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.140245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42393 (local address 127.0.0.1:57044)
0924 23:08:40.142255 (+ 2010us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.142304 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:40.142378 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:40.146213 (+ 3835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:40.146214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:40.146216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:40.146256 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.146257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.147452 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.147453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.148499 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.148502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.159075 (+ 10573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.159076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.159146 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.159151 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:08:40.159206 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:08:40.159251 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1974,"thread_start_us":77,"threads_started":1}
I20260924 23:08:40.159646 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.142034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.0.98.3:42393)
0924 23:08:40.145973 (+ 3939us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.145980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.145992 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:08:40.146027 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.146029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.146044 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.146127 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:40.146335 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.147358 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.158449 (+ 11091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.158937 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.159071 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.159136 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.159513 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.159567 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3898,"thread_start_us":63,"threads_started":1}
I20260924 23:08:40.160264 13522 tablet_service.cc:1467] Processing CreateTablet for tablet 9f72197fde394333ab74b44e678b4ebd (DEFAULT_TABLE table=test-table [id=91a2292bdecc4118b25f84b91cc64ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:40.160619 13522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f72197fde394333ab74b44e678b4ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:40.165068 13622 tablet_bootstrap.cc:492] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b: Bootstrap starting.
I20260924 23:08:40.165817 13622 tablet_bootstrap.cc:654] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:40.166206 13622 log.cc:824] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:40.166558 13623 tablet_bootstrap.cc:492] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Bootstrap starting.
I20260924 23:08:40.167661 13623 tablet_bootstrap.cc:654] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:40.168031 13623 log.cc:824] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:40.169056 13623 tablet_bootstrap.cc:492] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: No bootstrap required, opened a new log
I20260924 23:08:40.169062 13622 tablet_bootstrap.cc:492] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b: No bootstrap required, opened a new log
I20260924 23:08:40.169119 13623 ts_tablet_manager.cc:1397] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:08:40.169131 13622 ts_tablet_manager.cc:1397] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 23:08:40.169997 13621 tablet_bootstrap.cc:492] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a: Bootstrap starting.
I20260924 23:08:40.170701 13621 tablet_bootstrap.cc:654] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:40.171010 13621 log.cc:824] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:40.171903 13623 raft_consensus.cc:348] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.172096 13623 raft_consensus.cc:374] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:40.172132 13623 raft_consensus.cc:729] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f396619d26ef4129823e58a468a91aca, State: Initialized, Role: FOLLOWER
I20260924 23:08:40.172078 13622 raft_consensus.cc:348] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.172233 13622 raft_consensus.cc:374] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:40.172272 13622 raft_consensus.cc:729] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c26d8f27a4c42f29ff9110fd11ec01b, State: Initialized, Role: FOLLOWER
I20260924 23:08:40.172412 13622 consensus_queue.cc:260] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.172623 13623 consensus_queue.cc:260] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.172719 13622 ts_tablet_manager.cc:1428] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 23:08:40.172890 13320 heartbeater.cc:503] Master 127.0.98.62:33145 was elected leader, sending a full tablet report...
I20260924 23:08:40.172986 13623 ts_tablet_manager.cc:1428] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 23:08:40.173100 13588 heartbeater.cc:503] Master 127.0.98.62:33145 was elected leader, sending a full tablet report...
I20260924 23:08:40.174798 13621 tablet_bootstrap.cc:492] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a: No bootstrap required, opened a new log
I20260924 23:08:40.174872 13621 ts_tablet_manager.cc:1397] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 23:08:40.177502 13621 raft_consensus.cc:348] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.177673 13621 raft_consensus.cc:374] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:40.177707 13621 raft_consensus.cc:729] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc54a6e4d70d460888f321f47d94657a, State: Initialized, Role: FOLLOWER
I20260924 23:08:40.178267 13621 consensus_queue.cc:260] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.178558 13621 ts_tablet_manager.cc:1428] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 23:08:40.178728 13454 heartbeater.cc:503] Master 127.0.98.62:33145 was elected leader, sending a full tablet report...
I20260924 23:08:40.278832 13628 raft_consensus.cc:482] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:40.278975 13628 raft_consensus.cc:504] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.279400 13628 leader_election.cc:288] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c26d8f27a4c42f29ff9110fd11ec01b (127.0.98.1:35477), bc54a6e4d70d460888f321f47d94657a (127.0.98.2:44487)
I20260924 23:08:40.283684 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.279846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33183 (local address 127.0.98.1:35477)
0924 23:08:40.279892 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.279894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.280037 (+ 143us) server_negotiation.cc:407] Connection header received
0924 23:08:40.280192 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.280194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.280206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.280236 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:40.280456 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.281210 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.282717 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.283290 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.283338 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.283350 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.283572 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.283615 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:08:40.283710 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.279908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59575 (local address 127.0.98.2:44487)
0924 23:08:40.279961 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.279963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.280085 (+ 122us) server_negotiation.cc:407] Connection header received
0924 23:08:40.280186 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.280187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.280199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.280229 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:40.280452 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.281202 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.282726 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.283295 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.283344 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.283354 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.283593 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.283639 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:40.283932 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.279723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44487 (local address 127.0.98.3:59575)
0924 23:08:40.280023 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.280033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:40.280109 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:40.280295 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:40.280296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:40.280298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:40.280372 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.280374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.281310 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.281311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.282615 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.282617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.283430 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.283431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.283510 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.283514 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:08:40.283814 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 23:08:40.283853 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":60,"thread_start_us":80,"threads_started":1}
I20260924 23:08:40.284257 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.279594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35477 (local address 127.0.98.3:33183)
0924 23:08:40.279955 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.279966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:40.280067 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:40.280304 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:40.280305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:40.280307 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:40.280381 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.280383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.281314 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.281315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.282605 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:40.282608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.283421 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.283422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:40.283489 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.283494 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:08:40.284117 (+ 623us) client_negotiation.cc:240] Negotiation successful
0924 23:08:40.284160 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":130,"threads_started":1}
I20260924 23:08:40.284329 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72197fde394333ab74b44e678b4ebd" candidate_uuid: "f396619d26ef4129823e58a468a91aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc54a6e4d70d460888f321f47d94657a" is_pre_election: true
I20260924 23:08:40.284530 13408 raft_consensus.cc:2454] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f396619d26ef4129823e58a468a91aca in term 0.
I20260924 23:08:40.284545 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72197fde394333ab74b44e678b4ebd" candidate_uuid: "f396619d26ef4129823e58a468a91aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" is_pre_election: true
I20260924 23:08:40.284677 13271 raft_consensus.cc:2454] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f396619d26ef4129823e58a468a91aca in term 0.
I20260924 23:08:40.284785 13478 leader_election.cc:302] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc54a6e4d70d460888f321f47d94657a, f396619d26ef4129823e58a468a91aca; no voters:
I20260924 23:08:40.284963 13628 raft_consensus.cc:2781] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:08:40.285022 13628 raft_consensus.cc:482] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:08:40.285053 13628 raft_consensus.cc:3037] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:40.286015 13628 raft_consensus.cc:504] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.286154 13628 leader_election.cc:288] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [CANDIDATE]: Term 1 election: Requested vote from peers 2c26d8f27a4c42f29ff9110fd11ec01b (127.0.98.1:35477), bc54a6e4d70d460888f321f47d94657a (127.0.98.2:44487)
I20260924 23:08:40.286382 13408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72197fde394333ab74b44e678b4ebd" candidate_uuid: "f396619d26ef4129823e58a468a91aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc54a6e4d70d460888f321f47d94657a"
I20260924 23:08:40.286439 13271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f72197fde394333ab74b44e678b4ebd" candidate_uuid: "f396619d26ef4129823e58a468a91aca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b"
I20260924 23:08:40.286458 13408 raft_consensus.cc:3037] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:40.286525 13271 raft_consensus.cc:3037] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:40.287401 13408 raft_consensus.cc:2454] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f396619d26ef4129823e58a468a91aca in term 1.
I20260924 23:08:40.287484 13271 raft_consensus.cc:2454] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f396619d26ef4129823e58a468a91aca in term 1.
I20260924 23:08:40.287573 13478 leader_election.cc:302] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc54a6e4d70d460888f321f47d94657a, f396619d26ef4129823e58a468a91aca; no voters:
I20260924 23:08:40.287653 13628 raft_consensus.cc:2781] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:40.287789 13628 raft_consensus.cc:686] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 1 LEADER]: Becoming Leader. State: Replica: f396619d26ef4129823e58a468a91aca, State: Running, Role: LEADER
I20260924 23:08:40.287883 13628 consensus_queue.cc:237] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } }
I20260924 23:08:40.288702 13121 catalog_manager.cc:5495] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca reported cstate change: term changed from 0 to 1, leader changed from <none> to f396619d26ef4129823e58a468a91aca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f396619d26ef4129823e58a468a91aca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f396619d26ef4129823e58a468a91aca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 } health_report { overall_health: UNKNOWN } } }
I20260924 23:08:40.327913 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.324664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.0.98.3:42393)
0924 23:08:40.324711 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:40.324713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:40.324722 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:08:40.324771 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:40.324772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:40.324791 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:40.324861 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:08:40.325045 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.325841 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.326966 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:40.327280 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:40.327356 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:40.327535 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:08:40.327678 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:08:40.327765 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:40.327809 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 23:08:40.327832 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:08:40.330003 13271 raft_consensus.cc:1261] T 9f72197fde394333ab74b44e678b4ebd P 2c26d8f27a4c42f29ff9110fd11ec01b [term 1 FOLLOWER]: Refusing update from remote peer f396619d26ef4129823e58a468a91aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:40.330487 13632 consensus_queue.cc:1035] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c26d8f27a4c42f29ff9110fd11ec01b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:40.333562 13408 raft_consensus.cc:1261] T 9f72197fde394333ab74b44e678b4ebd P bc54a6e4d70d460888f321f47d94657a [term 1 FOLLOWER]: Refusing update from remote peer f396619d26ef4129823e58a468a91aca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:40.333902 13628 consensus_queue.cc:1035] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc54a6e4d70d460888f321f47d94657a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:40.337726 13636 mvcc.cc:205] Tried to move back new op lower bound from 7333033248068784128 to 7333033247899803648. Current Snapshot: MvccSnapshot[applied={T|T < 7333033248068784128}]
I20260924 23:08:40.338858 13121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:08:40.338953 13121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:08:40.339839 13121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 524f82d372244e3199d95014f4a14316: Sending DeleteTablet for 3 replicas of tablet 9f72197fde394333ab74b44e678b4ebd
I20260924 23:08:40.340269 13522 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f72197fde394333ab74b44e678b4ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57044
I20260924 23:08:40.340272 13388 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f72197fde394333ab74b44e678b4ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38696
I20260924 23:08:40.340705 13254 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f72197fde394333ab74b44e678b4ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47178
I20260924 23:08:40.340937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13160
I20260924 23:08:40.341563 13645 tablet_replica.cc:330] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: stopping tablet replica
I20260924 23:08:40.341697 13645 raft_consensus.cc:2229] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:08:40.345521 13645 raft_consensus.cc:2258] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:08:40.346287 13645 ts_tablet_manager.cc:1905] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 23:08:40.349380 13107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35477 (error 108)
I20260924 23:08:40.350013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13324
I20260924 23:08:40.358050 13645 ts_tablet_manager.cc:1918] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:08:40.358177 13645 log.cc:1196] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/9f72197fde394333ab74b44e678b4ebd
I20260924 23:08:40.358518 13645 ts_tablet_manager.cc:1939] T 9f72197fde394333ab74b44e678b4ebd P f396619d26ef4129823e58a468a91aca: Deleting consensus metadata
I20260924 23:08:40.358624 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.358452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35477 (local address 127.0.0.1:47180)
0924 23:08:40.358529 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.358577 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 23:08:40.358777 13107 catalog_manager.cc:4584] TS 2c26d8f27a4c42f29ff9110fd11ec01b (127.0.98.1:35477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f72197fde394333ab74b44e678b4ebd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35477: connect: Connection refused (error 111)
I20260924 23:08:40.359057 13106 catalog_manager.cc:4841] TS f396619d26ef4129823e58a468a91aca (127.0.98.3:42393): tablet 9f72197fde394333ab74b44e678b4ebd (table test-table [id=91a2292bdecc4118b25f84b91cc64ad6]) successfully deleted
I20260924 23:08:40.361631 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:40.361490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44487 (local address 127.0.0.1:38708)
0924 23:08:40.361575 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:40.361595 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 23:08:40.361774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13457
I20260924 23:08:40.368909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13089
I20260924 23:08:40.374625 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 23:08:40.708683 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 23:08:40.708777 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 23:08:41.053267 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 23:08:41.053 UTC [12156] LOG: received smart shutdown request
2026-09-24 23:08:41.056 UTC [12156] LOG: background worker "logical replication launcher" (PID 12165) exited with exit code 1
2026-09-24 23:08:41.057 UTC [12160] LOG: shutting down
2026-09-24 23:08:41.231 UTC [12156] LOG: database system is shut down
2026-09-24T23:08:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48175 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:08:41 dist-test-slave-ph9j krb5kdc[13658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:08:41 dist-test-slave-ph9j krb5kdc[13658](info): set up 2 sockets
Sep 24 23:08:41 dist-test-slave-ph9j krb5kdc[13658](info): commencing operation
krb5kdc: starting...
W20260924 23:08:42.377715 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:08:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:08:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:08:42.497399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43101
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43101,127.0.98.61:46269,127.0.98.60:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:42.581851 13671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:42.582142 13671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:42.582186 13671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:42.583384 13671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:42.583454 13671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:42.583479 13671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:42.583508 13671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:42.583559 13671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:42.585140 13671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43101,127.0.98.61:46269,127.0.98.60:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:42.585440 13671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:42.585685 13671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:42.590624 13677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:42.590821 13671 server_base.cc:1034] running on GCE node
W20260924 23:08:42.590657 13679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:42.590647 13676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:42.591140 13671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:42.591439 13671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:42.592605 13671 hybrid_clock.cc:648] HybridClock initialized: now 1790291322592585 us; error 47 us; skew 500 ppm
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:42.600796 13671 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:42.601294 13671 webserver.cc:460] Webserver started at http://127.0.98.62:32951/ using document root <none> and password file <none>
I20260924 23:08:42.601526 13671 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:42.601603 13671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:42.601754 13671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:42.602941 13671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "d03f48a09c7f49d39c3c1dedc6d28487"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "927f1524bc7887220cba1c11daf4e544"
server_key_iv: "e7deac0cc7449ea448a632a3bbaef40d"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.603313 13671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "d03f48a09c7f49d39c3c1dedc6d28487"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "927f1524bc7887220cba1c11daf4e544"
server_key_iv: "e7deac0cc7449ea448a632a3bbaef40d"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.604794 13671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:08:42.605589 13686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:42.605816 13671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:42.605938 13671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "d03f48a09c7f49d39c3c1dedc6d28487"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "927f1524bc7887220cba1c11daf4e544"
server_key_iv: "e7deac0cc7449ea448a632a3bbaef40d"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.606020 13671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:42.637748 13671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:42.640309 13671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:42.640487 13671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:42.645154 13671 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43101
I20260924 23:08:42.645205 13738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43101 every 8 connection(s)
I20260924 23:08:42.645588 13671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:42.646521 13739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:42.648118 13739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:42.654881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13671
I20260924 23:08:42.655052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:08:42.655278 392 external_mini_cluster.cc:1428] Setting key b8553f0e9652ad082690363bf0decf6e
I20260924 23:08:42.658416 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.648852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.0.98.62:43101)
0924 23:08:42.649087 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.649091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.649115 (+ 24us) server_negotiation.cc:407] Connection header received
0924 23:08:42.649599 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.649600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.649623 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.649698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.650260 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.651103 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.651949 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.652292 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.652376 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.654874 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.654881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.654881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.654888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.656654 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.657114 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.657117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.657117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.657201 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.657439 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.657440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.657440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.657663 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.657756 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.658009 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.658235 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":211,"spinlock_wait_cycles":8448,"thread_start_us":93,"threads_started":1}
I20260924 23:08:42.658421 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.648841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43101 (local address 127.0.0.1:42496)
0924 23:08:42.649010 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.649043 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.649464 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.649704 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.649706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.649969 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.650136 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.650141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.651189 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.651190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.651872 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.651874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.652489 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.652491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.652593 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.653105 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.653114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.654757 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.656778 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.656784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.656786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.657006 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.657272 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.657273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.657274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.657362 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.657735 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.657740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.657803 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.657934 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.658137 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260924 23:08:42.659389 13739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:08:42.659775 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.659656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37266)
0924 23:08:42.659713 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.659755 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 23:08:42.659917 13688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
I20260924 23:08:42.660374 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.660286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37280)
0924 23:08:42.660330 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.660353 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:08:42.660485 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 23:08:42.681088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46269
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43101,127.0.98.61:46269,127.0.98.60:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:42.709685 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } attempt: 1
I20260924 23:08:42.710210 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.710072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37296)
0924 23:08:42.710134 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.710177 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:08:42.710621 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.710532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37300)
0924 23:08:42.710578 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.710605 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:08:42.710738 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
W20260924 23:08:42.762367 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:42.762593 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:42.762615 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:42.763706 13746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:42.763739 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:42.763754 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:42.763767 13746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:42.763778 13746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:42.765208 13746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43101,127.0.98.61:46269,127.0.98.60:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:42.765435 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:42.765659 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:42.770344 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:42.770330 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:42.770363 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:42.770744 13746 server_base.cc:1034] running on GCE node
I20260924 23:08:42.770991 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:42.771324 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:42.772503 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790291322772480 us; error 39 us; skew 500 ppm
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:42.775079 13746 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 23:08:42.775555 13746 webserver.cc:460] Webserver started at http://127.0.98.61:45421/ using document root <none> and password file <none>
I20260924 23:08:42.775820 13746 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:42.775864 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:42.776008 13746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:42.777066 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data/instance:
uuid: "a1e161b7e7294d7cba30709780d83ead"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "3e22813ca746a9e8efe8c91f662ed49e"
server_key_iv: "660621bb301c058c1f5507def22f8dda"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.777429 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal/instance:
uuid: "a1e161b7e7294d7cba30709780d83ead"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "3e22813ca746a9e8efe8c91f662ed49e"
server_key_iv: "660621bb301c058c1f5507def22f8dda"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.778824 13746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:08:42.778904 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } attempt: 2
I20260924 23:08:42.779315 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.779168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37312)
0924 23:08:42.779236 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.779278 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 23:08:42.779750 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.779646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37316)
0924 23:08:42.779695 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.779723 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:08:42.779765 13761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260924 23:08:42.779878 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46269: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46269: connect: Connection refused (error 111)
I20260924 23:08:42.779928 13746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:42.780046 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal
uuid: "a1e161b7e7294d7cba30709780d83ead"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "3e22813ca746a9e8efe8c91f662ed49e"
server_key_iv: "660621bb301c058c1f5507def22f8dda"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.780113 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:42.810535 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:42.813144 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:42.813320 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:42.817770 13746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46269
I20260924 23:08:42.817811 13813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46269 every 8 connection(s)
I20260924 23:08:42.818204 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
I20260924 23:08:42.819128 13814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:42.820740 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:08:42.826469 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13746
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:42.826606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-1/wal/instance
I20260924 23:08:42.826792 392 external_mini_cluster.cc:1428] Setting key 1408ab168d6c83c2c5c2e3354c04feb4
I20260924 23:08:42.830636 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.821452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43101 (local address 127.0.0.1:42502)
0924 23:08:42.821641 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.821683 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.822161 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.822407 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.822410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.822655 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.822849 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.822853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.823549 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.823550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.824361 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.824363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.824741 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.824742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.824846 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.825341 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.825348 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.827123 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.829026 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.829034 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.829036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.829372 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.829649 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.829650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.829650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.829752 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.830097 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.830102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.830225 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.830313 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.830438 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":115,"threads_started":1}
I20260924 23:08:42.830641 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.821473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.0.98.62:43101)
0924 23:08:42.821593 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.821594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.821778 (+ 184us) server_negotiation.cc:407] Connection header received
0924 23:08:42.822254 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.822255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.822268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.822334 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.822925 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.823460 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.824438 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.824667 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.824725 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.827241 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.827243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.827243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.827252 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.828909 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.829473 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.829474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.829474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.829578 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.829822 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.829823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.829824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.830023 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.830100 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.830363 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.830469 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 23:08:42.831368 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:08:42.841706 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.831684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37322)
0924 23:08:42.831761 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.831774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.831822 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.832109 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.832110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.832264 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.832307 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.832309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.833207 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.833208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.833738 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.833740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.834268 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.834269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.834339 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.835636 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.835648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.838535 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.840542 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.840543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.840544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.840714 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.840939 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.840939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.840939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.840992 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.841335 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.841336 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.841374 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.841429 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.841537 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 23:08:42.841826 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.831798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.0.98.61:46269)
0924 23:08:42.831978 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.831982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.831997 (+ 15us) server_negotiation.cc:407] Connection header received
0924 23:08:42.832014 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.832016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.832026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.832109 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.832400 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.833111 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.833836 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.834178 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.834269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.838625 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.838634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.838635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.838643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.840454 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.840798 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.840802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.840802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.840869 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.841064 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.841065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.841065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.841268 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.841354 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.841465 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.841617 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260924 23:08:42.842512 13814 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:08:42.842911 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.842782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33862)
0924 23:08:42.842835 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.842891 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 23:08:42.843048 13765 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
I20260924 23:08:42.843511 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.843407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33866)
0924 23:08:42.843458 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.843483 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 23:08:42.843626 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38837: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 23:08:42.853011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38837
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43101,127.0.98.61:46269,127.0.98.60:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:42.871035 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } attempt: 3
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:08:42.878237 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.871374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37328)
0924 23:08:42.871437 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.871450 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.871520 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.871694 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.871695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.871856 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.871918 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.871921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.872613 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.872614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.873136 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.873138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.873553 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.873554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.873637 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.874160 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.874173 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.875574 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.877173 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.877174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.877174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.877345 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.877555 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.877556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.877556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.877617 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.877868 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.877869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.877945 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.878031 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.878138 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 23:08:42.878237 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.871468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.98.61:46269)
0924 23:08:42.871537 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.871539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.871549 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:08:42.871606 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.871607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.871619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.871687 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.872005 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.872513 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.873222 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.873445 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.873553 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.875666 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.875668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.875668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.875676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.877069 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.877429 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.877431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.877431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.877488 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.877683 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.877684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.877685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.877801 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.877876 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.878056 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.878140 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:08:42.878793 13739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:08:42.879148 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.879031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33876)
0924 23:08:42.879097 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.879127 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 23:08:42.879473 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.879396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33886)
0924 23:08:42.879437 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.879459 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:08:42.879576 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38837: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
I20260924 23:08:42.892858 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } attempt: 1
I20260924 23:08:42.893270 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.893146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33890)
0924 23:08:42.893205 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.893241 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:08:42.893921 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.893729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33900)
0924 23:08:42.893809 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.893849 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 23:08:42.894078 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38837: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
I20260924 23:08:42.910714 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } attempt: 1
I20260924 23:08:42.911162 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.911028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33908)
0924 23:08:42.911096 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.911134 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 23:08:42.911505 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.911405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33920)
0924 23:08:42.911450 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.911477 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:08:42.911594 13739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38837: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
W20260924 23:08:42.935087 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:42.935283 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:42.935304 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:42.936422 13821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:42.936467 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:42.936481 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:42.936493 13821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:42.936506 13821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:42.938095 13821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43101,127.0.98.61:46269,127.0.98.60:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:42.938337 13821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:42.938592 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:42.943161 13827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:42.943215 13826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:42.943176 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:42.943403 13821 server_base.cc:1034] running on GCE node
I20260924 23:08:42.943662 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:42.943934 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:42.945088 13821 hybrid_clock.cc:648] HybridClock initialized: now 1790291322945064 us; error 36 us; skew 500 ppm
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:42.947638 13821 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 23:08:42.948087 13821 webserver.cc:460] Webserver started at http://127.0.98.60:39387/ using document root <none> and password file <none>
I20260924 23:08:42.948313 13821 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:42.948382 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:42.948540 13821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:42.949586 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data/instance:
uuid: "589b1a839950419bad3397d8fc7ea373"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "80aedbaa1279273301726dcfb5c2bc95"
server_key_iv: "eb2664ed9865d8ef00236e32f545dc5b"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.950001 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal/instance:
uuid: "589b1a839950419bad3397d8fc7ea373"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "80aedbaa1279273301726dcfb5c2bc95"
server_key_iv: "eb2664ed9865d8ef00236e32f545dc5b"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.951282 13821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:42.952010 13836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:42.952186 13821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:08:42.952280 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal
uuid: "589b1a839950419bad3397d8fc7ea373"
format_stamp: "Formatted at 2026-09-24 23:08:42 on dist-test-slave-ph9j"
server_key: "80aedbaa1279273301726dcfb5c2bc95"
server_key_iv: "eb2664ed9865d8ef00236e32f545dc5b"
server_key_version: "encryptionkey@0"
I20260924 23:08:42.952366 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:42.962230 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } attempt: 2
I20260924 23:08:42.962642 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.962489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33924)
0924 23:08:42.962556 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.962600 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 23:08:42.963277 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.963174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33926)
0924 23:08:42.963220 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.963251 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:08:42.963377 13814 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38837: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38837: connect: Connection refused (error 111)
I20260924 23:08:42.964171 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:42.966085 13821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:42.966193 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:42.970079 13821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38837
I20260924 23:08:42.970247 13888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38837 every 8 connection(s)
I20260924 23:08:42.970487 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
I20260924 23:08:42.971294 13889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:42.972847 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:08:42.977653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13821
I20260924 23:08:42.977769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790291169445806-392-0/minicluster-data/master-2/wal/instance
I20260924 23:08:42.977988 392 external_mini_cluster.cc:1428] Setting key aa84f18038530d192b5847e59fe896bf
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:42.982501 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.973590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.0.98.62:43101)
0924 23:08:42.973675 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.973677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.973865 (+ 188us) server_negotiation.cc:407] Connection header received
0924 23:08:42.974359 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.974360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.974371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.974436 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.975027 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.975498 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.976455 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.976658 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.976718 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.979391 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.979392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.979393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.979399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.980930 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.981388 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.981389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.981390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.981436 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.981700 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.981701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.981702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.981828 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.981919 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.982312 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.982397 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 23:08:42.982702 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.973555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43101 (local address 127.0.0.1:42516)
0924 23:08:42.973717 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.973756 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.974254 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.974488 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.974492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.974738 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.974923 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.974927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.975608 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.975608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.976377 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.976379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.976734 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.976735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.976834 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.977356 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.977367 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.979292 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.981012 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.981019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.981022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.981314 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.981525 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.981526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.981526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.981637 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.981938 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.981943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.982159 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.982278 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.982445 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260924 23:08:42.983325 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:08:42.986714 13739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } attempt: 2
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:08:42.988409 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.978380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.0.98.60:38837)
0924 23:08:42.978509 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.978513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.978611 (+ 98us) server_negotiation.cc:407] Connection header received
0924 23:08:42.978668 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.978669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.978685 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.978745 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.979482 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.980575 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.981492 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.981953 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.982017 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.984701 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.984709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.984710 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.984717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.986616 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.987178 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.987181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.987182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.987249 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.987505 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.987506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.987506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.987692 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.987853 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.988208 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.988299 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260924 23:08:42.991642 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.983623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46269 (local address 127.0.0.1:37338)
0924 23:08:42.984260 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.984271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.984322 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.984441 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.984442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.984587 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.984645 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.984646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.985457 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.985458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.986173 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.986175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.986717 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.986718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.986812 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.987231 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.987236 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.988661 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.990486 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.990488 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.990488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.990731 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.990950 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.990951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.990951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.991015 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.991313 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.991314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.991360 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.991431 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.991532 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620}
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:08:42.992019 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.983663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.98.61:46269)
0924 23:08:42.983732 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.983734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.984309 (+ 575us) server_negotiation.cc:407] Connection header received
0924 23:08:42.984380 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.984381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.984390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.984435 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.984716 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.985359 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.986257 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.986582 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.986662 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.988763 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.988764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.988765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.988772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.990403 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.990812 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.990814 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.990814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.990875 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.991089 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.991090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.991090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.991249 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.991321 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.991831 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.991922 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 23:08:42.992373 13889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:42.996665 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.986993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33948)
0924 23:08:42.987043 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.987055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.987131 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.987633 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.987635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.987831 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.987876 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.987878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.988764 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.988765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.989343 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.989345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.990141 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.990142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.990210 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.990602 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.990608 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.992092 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:42.995522 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.995523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.995523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.995758 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.996013 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:42.996014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:42.996014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:42.996084 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:42.996348 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:42.996349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:42.996392 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:08:42.996456 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:08:42.996546 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 23:08:42.997001 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.987240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.0.98.60:38837)
0924 23:08:42.987442 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.987444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.987451 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:08:42.987527 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.987533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.987543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.987629 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.987945 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.988686 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.989430 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.990055 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.990142 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.992255 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.992256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.992256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.992263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.995451 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.995845 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.995846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.995847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.995933 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.996154 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.996155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.996156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.996289 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.996355 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.996786 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.996860 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":118,"threads_started":1}
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:08:42.998548 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.989775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42530 (local address 127.0.98.62:43101)
0924 23:08:42.989816 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.989818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.989865 (+ 47us) server_negotiation.cc:407] Connection header received
0924 23:08:42.989987 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.989988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.989998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.990058 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.990704 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.991351 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.992358 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.992675 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.992724 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.995405 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.995407 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.995407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.995415 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:42.997264 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.997643 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.997645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.997645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.997702 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:42.997934 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:42.997936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:42.997936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:42.998045 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:42.998368 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:42.998424 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 23:08:42.998493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:42.999121 13739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: Bootstrap starting.
I20260924 23:08:42.999890 13739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:43.000234 13739 log.cc:824] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:43.001122 13739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: No bootstrap required, opened a new log
I20260924 23:08:43.001479 13891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.992593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33964)
0924 23:08:42.993852 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:42.993863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:42.993928 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:42.994329 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:42.994331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:42.994536 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:42.994566 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.994567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.995477 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.995478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.996180 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:42.996182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.996639 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.996640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:42.996690 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.997312 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:42.997317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:42.998707 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:43.000352 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:43.000353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:43.000353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:43.000525 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:43.000771 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:43.000771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:43.000772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:43.000823 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:43.001134 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:43.001135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:43.001177 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 23:08:43.001271 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 23:08:43.001361 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243}
I20260924 23:08:43.001813 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:42.993788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.0.98.60:38837)
0924 23:08:42.994200 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:42.994202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:42.994210 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:08:42.994230 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:42.994231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:42.994241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:42.994304 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:42.994648 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.995369 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.996251 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:42.996541 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:42.996640 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:42.998793 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:42.998793 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:42.998794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:42.998800 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:43.000269 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.000651 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.000652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.000653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.000704 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.000895 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.000896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.000897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.001054 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:43.001128 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:43.001315 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 23:08:43.001420 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":393}
I20260924 23:08:43.003014 13739 raft_consensus.cc:348] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.003165 13739 raft_consensus.cc:374] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:43.003211 13739 raft_consensus.cc:729] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03f48a09c7f49d39c3c1dedc6d28487, State: Initialized, Role: FOLLOWER
I20260924 23:08:43.003268 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: Bootstrap starting.
I20260924 23:08:43.003360 13739 consensus_queue.cc:260] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.003647 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.003705 13739 sys_catalog.cc:564] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:43.003786 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:08:43.004321 13889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:43.004642 13889 log.cc:824] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: Log is configured to *not* fsync() on all Append() calls
Sep 24 23:08:42 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:08:43.005465 13889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: No bootstrap required, opened a new log
W20260924 23:08:43.005527 13908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:08:43.005584 13908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:08:43.007642 13889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.007774 13889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:43.007814 13889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589b1a839950419bad3397d8fc7ea373, State: Initialized, Role: FOLLOWER
I20260924 23:08:43.007920 13889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.008177 13889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:43.009622 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:43.000574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.61:46269)
0924 23:08:43.000615 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:43.000617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:43.000626 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:08:43.000648 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:43.000649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:43.000659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:43.000719 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:43.001259 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.001959 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.002808 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.003085 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.003163 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:43.005721 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:43.005722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:43.005722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:43.005730 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:43.008020 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.008464 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.008465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.008466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.008524 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.008737 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.008738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.008738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.008864 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:43.009016 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:43.009421 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 23:08:43.009516 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:08:43.006348 13671 master_runner.cc:186] Error getting master registration for 127.0.98.61:46269: OK, instance_id { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" instance_seqno: 1790291322816491 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:08:43.009775 13671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:08:43.009934 13910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.010043 13910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:08:43.010761 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 23:08:43.010808 13921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:08:43.010865 13921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:08:43.011420 13821 master_runner.cc:186] Error getting master registration for 127.0.98.61:46269: OK, instance_id { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" instance_seqno: 1790291322816491 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:08:43.013619 13821 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:08:43.020632 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:43.012168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.0.98.62:43101)
0924 23:08:43.012204 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:43.012206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:43.013028 (+ 822us) server_negotiation.cc:407] Connection header received
0924 23:08:43.013122 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:43.013123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:43.013133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:43.013164 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:43.013937 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.014663 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.015727 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.016031 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.016107 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:43.017554 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:43.017555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:43.017556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:43.017562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:43.019385 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.019785 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.019787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.019787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.019847 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.020062 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.020062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.020063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.020187 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:43.020254 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:43.020455 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 23:08:43.020538 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:08:43.025429 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:43.012467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.98.60:38837)
0924 23:08:43.012508 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:43.012510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:43.018035 (+ 5525us) server_negotiation.cc:407] Connection header received
0924 23:08:43.018111 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:43.018112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:43.018124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:43.018177 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:43.018877 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.019574 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.020597 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.020887 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.020925 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:43.022651 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:43.022652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:43.022653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:43.022660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:43.024277 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.024654 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.024655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.024655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.024704 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.024887 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.024888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.024888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.025003 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:43.025066 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:43.025258 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 23:08:43.025336 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:08:43.028946 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:43.012790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.0.98.61:46269)
0924 23:08:43.012823 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:43.012825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:43.018200 (+ 5375us) server_negotiation.cc:407] Connection header received
0924 23:08:43.018292 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:43.018293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:43.018303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:43.018333 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:43.022005 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.022708 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.024339 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.024622 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.024662 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:43.026078 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:43.026079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:43.026079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:43.026086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:43.027719 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.028123 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.028125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.028125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.028187 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.028404 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.028405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.028405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.028533 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:43.028634 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:43.028775 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 23:08:43.028853 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:08:43.054534 13814 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } attempt: 3
Sep 24 23:08:43 dist-test-slave-ph9j krb5kdc[13658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:08:43.063418 13816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:43.054850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38837 (local address 127.0.0.1:33982)
0924 23:08:43.054910 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:43.054925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:43.054973 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:43.055162 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:43.055163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:43.055389 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:43.055455 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:43.055457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.056512 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.056513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:43.057416 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:43.057419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.057973 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.057974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:43.058044 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:43.058564 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:43.058570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:43.060230 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:43.062293 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:43.062294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:43.062294 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:43.062515 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:43.062759 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:43.062760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:43.062761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:43.062821 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:43.063110 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:43.063111 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:43.063154 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:08:43.063215 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:08:43.063311 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 23:08:43.063786 13893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:43.054918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.0.98.60:38837)
0924 23:08:43.054972 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:43.054974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:43.055023 (+ 49us) server_negotiation.cc:407] Connection header received
0924 23:08:43.055047 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:43.055048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:43.055058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:43.055118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:43.055619 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.056422 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.057543 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:43.057852 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:43.057956 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:43.060324 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:43.060325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:43.060326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:43.060334 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:43.062192 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.062623 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.062625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.062625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.062683 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:43.062885 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:43.062886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:43.062886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:43.063025 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:43.063084 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:43.063614 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 23:08:43.063688 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:08:43.065433 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: Bootstrap starting.
I20260924 23:08:43.066227 13814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:43.066529 13814 log.cc:824] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:43.067440 13814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: No bootstrap required, opened a new log
I20260924 23:08:43.069444 13814 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.069593 13814 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:43.069628 13814 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e161b7e7294d7cba30709780d83ead, State: Initialized, Role: FOLLOWER
I20260924 23:08:43.069751 13814 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.070154 13814 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:43.070120 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.070320 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:08:43.072438 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:08:43.072494 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:08:43.074047 13746 master_runner.cc:263] No leader master found from master a1e161b7e7294d7cba30709780d83ead
I20260924 23:08:43.074098 13746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 23:08:43.141350 13692 tablet.cc:2369] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:08:43.216702 13842 tablet.cc:2369] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:08:43.278100 13910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:43.278242 13910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.278611 13910 leader_election.cc:288] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d03f48a09c7f49d39c3c1dedc6d28487 (127.0.98.62:43101), a1e161b7e7294d7cba30709780d83ead (127.0.98.61:46269)
I20260924 23:08:43.278949 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "589b1a839950419bad3397d8fc7ea373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" is_pre_election: true
I20260924 23:08:43.278997 13788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "589b1a839950419bad3397d8fc7ea373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e161b7e7294d7cba30709780d83ead" is_pre_election: true
I20260924 23:08:43.279155 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589b1a839950419bad3397d8fc7ea373 in term 0.
I20260924 23:08:43.279157 13788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 589b1a839950419bad3397d8fc7ea373 in term 0.
I20260924 23:08:43.279381 13838 leader_election.cc:302] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589b1a839950419bad3397d8fc7ea373, a1e161b7e7294d7cba30709780d83ead; no voters:
I20260924 23:08:43.279539 13910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:08:43.279613 13910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:08:43.279637 13910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:43.280369 13910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.280507 13910 leader_election.cc:288] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [CANDIDATE]: Term 1 election: Requested vote from peers d03f48a09c7f49d39c3c1dedc6d28487 (127.0.98.62:43101), a1e161b7e7294d7cba30709780d83ead (127.0.98.61:46269)
I20260924 23:08:43.280656 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "589b1a839950419bad3397d8fc7ea373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03f48a09c7f49d39c3c1dedc6d28487"
I20260924 23:08:43.280741 13713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:43.280740 13788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "589b1a839950419bad3397d8fc7ea373" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e161b7e7294d7cba30709780d83ead"
I20260924 23:08:43.280859 13788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:43.281533 13713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589b1a839950419bad3397d8fc7ea373 in term 1.
I20260924 23:08:43.281698 13788 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 589b1a839950419bad3397d8fc7ea373 in term 1.
I20260924 23:08:43.281710 13840 leader_election.cc:302] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 589b1a839950419bad3397d8fc7ea373, d03f48a09c7f49d39c3c1dedc6d28487; no voters:
I20260924 23:08:43.281862 13910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:43.282055 13910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [term 1 LEADER]: Becoming Leader. State: Replica: 589b1a839950419bad3397d8fc7ea373, State: Running, Role: LEADER
I20260924 23:08:43.282171 13910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } }
I20260924 23:08:43.282572 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589b1a839950419bad3397d8fc7ea373. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.282665 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:43.282846 13946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:43.283020 13946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:43.284329 13788 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [term 1 FOLLOWER]: Refusing update from remote peer 589b1a839950419bad3397d8fc7ea373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:43.284359 13713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [term 1 FOLLOWER]: Refusing update from remote peer 589b1a839950419bad3397d8fc7ea373: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:43.284649 13944 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:43.284781 13910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:43.285851 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589b1a839950419bad3397d8fc7ea373. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.285997 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:08:43.286075 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589b1a839950419bad3397d8fc7ea373. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.286159 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:08:43.286818 13947 mvcc.cc:205] Tried to move back new op lower bound from 7333033260170297344 to 7333033260164296704. Current Snapshot: MvccSnapshot[applied={T|T < 7333033260170297344}]
I20260924 23:08:43.286890 13910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.286971 13910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:43.287015 13910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.287050 13910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:43.287056 13931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.287101 13896 sys_catalog.cc:455] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "589b1a839950419bad3397d8fc7ea373" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03f48a09c7f49d39c3c1dedc6d28487" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43101 } } peers { permanent_uuid: "a1e161b7e7294d7cba30709780d83ead" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46269 } } peers { permanent_uuid: "589b1a839950419bad3397d8fc7ea373" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38837 } } }
I20260924 23:08:43.287163 13896 sys_catalog.cc:458] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:08:43.287164 13931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:08:43.287285 13948 mvcc.cc:205] Tried to move back new op lower bound from 7333033260170297344 to 7333033260164296704. Current Snapshot: MvccSnapshot[applied={T|T < 7333033260170297344}]
I20260924 23:08:43.287423 13949 mvcc.cc:205] Tried to move back new op lower bound from 7333033260170297344 to 7333033260164296704. Current Snapshot: MvccSnapshot[applied={T|T < 7333033260170297344}]
I20260924 23:08:43.287628 13946 catalog_manager.cc:1347] Generated new cluster ID: 488a3348a5024be496ee44ddb904d8b1
I20260924 23:08:43.287699 13946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:08:43.311187 13946 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:08:43.311868 13946 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:08:43.325652 13946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 589b1a839950419bad3397d8fc7ea373: Generated new TSK 0
I20260924 23:08:43.325918 13946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:08:44.006021 13908 catalog_manager.cc:1259] Loaded cluster ID: 488a3348a5024be496ee44ddb904d8b1
I20260924 23:08:44.006122 13908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: loading cluster ID for follower catalog manager: success
I20260924 23:08:44.006928 13908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: acquiring CA information for follower catalog manager: success
I20260924 23:08:44.007251 13908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d03f48a09c7f49d39c3c1dedc6d28487: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:08:44.072921 13942 catalog_manager.cc:1259] Loaded cluster ID: 488a3348a5024be496ee44ddb904d8b1
I20260924 23:08:44.073004 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: loading cluster ID for follower catalog manager: success
I20260924 23:08:44.073858 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: acquiring CA information for follower catalog manager: success
I20260924 23:08:44.074239 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a1e161b7e7294d7cba30709780d83ead: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:08:44.383044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13671
I20260924 23:08:44.388273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13746
I20260924 23:08:44.393043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13821
2026-09-24T23:08:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3051 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:08:44 dist-test-slave-ph9j krb5kdc[13962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:08:44 dist-test-slave-ph9j krb5kdc[13962](info): set up 2 sockets
Sep 24 23:08:44 dist-test-slave-ph9j krb5kdc[13962](info): commencing operation
krb5kdc: starting...
W20260924 23:08:45.425789 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:08:45 dist-test-slave-ph9j krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:08:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:08:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:08:45.540861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42619
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:45.619352 13975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:45.619608 13975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:45.619647 13975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:45.620827 13975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:45.620903 13975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:45.620929 13975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:45.620957 13975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:45.620990 13975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:45.622495 13975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:45.622779 13975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:45.623047 13975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:08:45.627892 13975 server_base.cc:1034] running on GCE node
W20260924 23:08:45.627876 13983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:45.627882 13981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:45.627875 13980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:45.628427 13975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:45.628722 13975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:45.629863 13975 hybrid_clock.cc:648] HybridClock initialized: now 1790291325629845 us; error 38 us; skew 500 ppm
Sep 24 23:08:45 dist-test-slave-ph9j krb5kdc[13962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:45.632287 13975 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:45.632764 13975 webserver.cc:460] Webserver started at http://127.0.98.62:44893/ using document root <none> and password file <none>
I20260924 23:08:45.633011 13975 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:45.633105 13975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:45.633262 13975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:45.634394 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "42689e0cfd024644a6e3e1dc61a9f85b"
format_stamp: "Formatted at 2026-09-24 23:08:45 on dist-test-slave-ph9j"
server_key: "9b85c381b9ba9d5f21b892989b7a3a6f"
server_key_iv: "82ebe59b1d081cf7e10983e2a6ed860f"
server_key_version: "encryptionkey@0"
I20260924 23:08:45.634753 13975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "42689e0cfd024644a6e3e1dc61a9f85b"
format_stamp: "Formatted at 2026-09-24 23:08:45 on dist-test-slave-ph9j"
server_key: "9b85c381b9ba9d5f21b892989b7a3a6f"
server_key_iv: "82ebe59b1d081cf7e10983e2a6ed860f"
server_key_version: "encryptionkey@0"
I20260924 23:08:45.636047 13975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:08:45.636793 13990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:45.636960 13975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:08:45.637077 13975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "42689e0cfd024644a6e3e1dc61a9f85b"
format_stamp: "Formatted at 2026-09-24 23:08:45 on dist-test-slave-ph9j"
server_key: "9b85c381b9ba9d5f21b892989b7a3a6f"
server_key_iv: "82ebe59b1d081cf7e10983e2a6ed860f"
server_key_version: "encryptionkey@0"
I20260924 23:08:45.637157 13975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:45.657284 13975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:45.659272 13975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:45.659443 13975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:45.663543 13975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42619
I20260924 23:08:45.663578 14042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42619 every 8 connection(s)
I20260924 23:08:45.663928 13975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:45.664758 14043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:45.665405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 13975
I20260924 23:08:45.665511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:08:45.665711 392 external_mini_cluster.cc:1428] Setting key b1afe9ab9390b7750b92b8b2b1501045
I20260924 23:08:45.667850 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b: Bootstrap starting.
I20260924 23:08:45.668452 14043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:45.668746 14043 log.cc:824] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:45.669508 14043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b: No bootstrap required, opened a new log
I20260924 23:08:45.672022 14043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42619 } }
I20260924 23:08:45.672211 14043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:45.672237 14043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42689e0cfd024644a6e3e1dc61a9f85b, State: Initialized, Role: FOLLOWER
I20260924 23:08:45.672380 14043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42619 } }
I20260924 23:08:45.672456 14043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:08:45.672478 14043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:08:45.672544 14043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 0 FOLLOWER]: Advancing to term 1
Sep 24 23:08:45 dist-test-slave-ph9j krb5kdc[13962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:45.673115 14043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42619 } }
I20260924 23:08:45.673237 14043 leader_election.cc:302] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42689e0cfd024644a6e3e1dc61a9f85b; no voters:
I20260924 23:08:45.673437 14043 leader_election.cc:288] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:08:45.673473 14048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:45.673588 14048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [term 1 LEADER]: Becoming Leader. State: Replica: 42689e0cfd024644a6e3e1dc61a9f85b, State: Running, Role: LEADER
I20260924 23:08:45.673718 14043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:45.673738 14048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42619 } }
I20260924 23:08:45.674086 14048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42689e0cfd024644a6e3e1dc61a9f85b. Latest consensus state: current_term: 1 leader_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42619 } } }
I20260924 23:08:45.674098 14049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42689e0cfd024644a6e3e1dc61a9f85b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42619 } } }
I20260924 23:08:45.674243 14048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:45.674737 14049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b [sys.catalog]: This master's current role is: LEADER
W20260924 23:08:45.675594 14064 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:08:45.675657 14064 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:08:45.675707 14057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:45.675870 14057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:45.677119 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:45.666250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.0.98.62:42619)
0924 23:08:45.666432 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:45.666441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:45.666477 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:08:45.666893 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:45.666900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:45.666945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:45.667059 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:45.667847 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:45.668889 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:45.669940 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:45.670335 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:45.670423 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:45.673306 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:45.673310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:45.673310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:45.673346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:45.675484 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:45.675918 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:45.675921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:45.675922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:45.676024 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:45.676253 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:45.676254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:45.676255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:45.676492 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:45.676632 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:45.676757 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 23:08:45.676892 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260924 23:08:45.677338 14057 catalog_manager.cc:1347] Generated new cluster ID: 9c184256711c47e7828177bb672a7e3c
I20260924 23:08:45.677389 14057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:08:45.678056 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:08:45.687690 14057 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:08:45.688400 14057 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:08:45.695097 14057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42689e0cfd024644a6e3e1dc61a9f85b: Generated new TSK 0
I20260924 23:08:45.695303 14057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:08:45.731932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 13975
2026-09-24T23:08:45Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1337 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:08:45 dist-test-slave-ph9j krb5kdc[14070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:08:45 dist-test-slave-ph9j krb5kdc[14070](info): set up 2 sockets
Sep 24 23:08:45 dist-test-slave-ph9j krb5kdc[14070](info): commencing operation
krb5kdc: starting...
W20260924 23:08:46.774017 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:08:46 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:08:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:08:46.892796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:46.974395 14083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:46.974630 14083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:46.974654 14083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:46.975875 14083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:46.975922 14083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:46.975937 14083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:46.975950 14083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:46.975961 14083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:46.977567 14083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:46.977836 14083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:46.978092 14083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:46.983179 14088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:46.983265 14091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:46.983198 14083 server_base.cc:1034] running on GCE node
W20260924 23:08:46.983311 14089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:46.983760 14083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:46.984025 14083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:46.985196 14083 hybrid_clock.cc:648] HybridClock initialized: now 1790291326985187 us; error 38 us; skew 500 ppm
Sep 24 23:08:46 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:46.987723 14083 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:46.988180 14083 webserver.cc:460] Webserver started at http://127.0.98.62:37755/ using document root <none> and password file <none>
I20260924 23:08:46.988421 14083 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:46.988485 14083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:46.988636 14083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:46.989699 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "fdbe0481abb7416c876492d285e91656"
format_stamp: "Formatted at 2026-09-24 23:08:46 on dist-test-slave-ph9j"
server_key: "115036317581b2783b954798bfab0ba3"
server_key_iv: "0ee4c218b14d2c056bfa0a5605c9a04e"
server_key_version: "encryptionkey@0"
I20260924 23:08:46.990093 14083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdbe0481abb7416c876492d285e91656"
format_stamp: "Formatted at 2026-09-24 23:08:46 on dist-test-slave-ph9j"
server_key: "115036317581b2783b954798bfab0ba3"
server_key_iv: "0ee4c218b14d2c056bfa0a5605c9a04e"
server_key_version: "encryptionkey@0"
I20260924 23:08:46.991382 14083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:46.992089 14098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:46.992273 14083 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:08:46.992373 14083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "fdbe0481abb7416c876492d285e91656"
format_stamp: "Formatted at 2026-09-24 23:08:46 on dist-test-slave-ph9j"
server_key: "115036317581b2783b954798bfab0ba3"
server_key_iv: "0ee4c218b14d2c056bfa0a5605c9a04e"
server_key_version: "encryptionkey@0"
I20260924 23:08:46.992448 14083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:47.002274 14083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:47.004153 14083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:47.004315 14083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:47.008307 14083 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37419
I20260924 23:08:47.008343 14150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37419 every 8 connection(s)
I20260924 23:08:47.008729 14083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:47.009773 14151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:47.012117 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap starting.
I20260924 23:08:47.012704 14151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:47.012988 14151 log.cc:824] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:47.013695 14151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: No bootstrap required, opened a new log
I20260924 23:08:47.015372 14151 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.015512 14151 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:47.015534 14151 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbe0481abb7416c876492d285e91656, State: Initialized, Role: FOLLOWER
I20260924 23:08:47.015708 14151 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.015794 14151 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:08:47.015848 14151 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:08:47.015900 14151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:47.016722 14151 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.016837 14151 leader_election.cc:302] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdbe0481abb7416c876492d285e91656; no voters:
I20260924 23:08:47.016995 14151 leader_election.cc:288] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:08:47.017023 14154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:47.017216 14154 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 LEADER]: Becoming Leader. State: Replica: fdbe0481abb7416c876492d285e91656, State: Running, Role: LEADER
I20260924 23:08:47.017263 14151 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:47.017333 14154 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.017740 14156 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdbe0481abb7416c876492d285e91656. Latest consensus state: current_term: 1 leader_uuid: "fdbe0481abb7416c876492d285e91656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260924 23:08:47.017731 14155 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdbe0481abb7416c876492d285e91656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260924 23:08:47.017797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14083
I20260924 23:08:47.017832 14156 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:47.017840 14155 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:47.017943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:08:47.018051 14161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:47.018190 392 external_mini_cluster.cc:1428] Setting key 3b7a1c1b5fab985211bf6db295812189
I20260924 23:08:47.018548 14161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:47.020185 14161 catalog_manager.cc:1347] Generated new cluster ID: e1253519c5d54a24b9671a6b74c2f608
I20260924 23:08:47.020239 14161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:47.031190 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.018852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.0.98.62:37419)
0924 23:08:47.022026 (+ 3174us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.022034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.022065 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:08:47.022169 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.022174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.022232 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.022377 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:47.023009 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.023956 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.024747 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.025223 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.025350 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.027775 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:47.027779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:47.027779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:47.027791 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:47.029546 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.029953 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.029956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.029956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.030047 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.030270 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.030271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.030271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.030533 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:47.030707 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.030803 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.030950 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":3106,"thread_start_us":96,"threads_started":1}
I20260924 23:08:47.038941 14161 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:08:47.039695 14161 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:08:47.047294 14161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Generated new TSK 0
I20260924 23:08:47.047585 14161 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:08:47.081586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:47.163715 14177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:47.163976 14177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:47.164013 14177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:47.165162 14177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:47.165235 14177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:47.165279 14177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:08:47.166776 14177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:47.167054 14177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:47.167282 14177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:47.172719 14183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.172757 14182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.172793 14177 server_base.cc:1034] running on GCE node
W20260924 23:08:47.172757 14185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.173130 14177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:47.173349 14177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:47.174489 14177 hybrid_clock.cc:648] HybridClock initialized: now 1790291327174476 us; error 44 us; skew 500 ppm
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:47.177029 14177 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:08:47.177461 14177 webserver.cc:460] Webserver started at http://127.0.98.1:42259/ using document root <none> and password file <none>
I20260924 23:08:47.177717 14177 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:47.177764 14177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:47.177934 14177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:47.179054 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "a204e31364a94edfb6e5c8b82de4927a"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8192d020fb1892483887375734b2f675"
server_key_iv: "4756ed7ab72678d4d444955ac43f7c40"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.179458 14177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a204e31364a94edfb6e5c8b82de4927a"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8192d020fb1892483887375734b2f675"
server_key_iv: "4756ed7ab72678d4d444955ac43f7c40"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.180706 14177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:08:47.181418 14192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.181648 14177 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:47.181751 14177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "a204e31364a94edfb6e5c8b82de4927a"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8192d020fb1892483887375734b2f675"
server_key_iv: "4756ed7ab72678d4d444955ac43f7c40"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.181829 14177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:47.216782 14177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:47.222597 14177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:47.222770 14177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:47.223031 14177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:47.223387 14177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:47.223421 14177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.223445 14177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:47.223491 14177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.229244 14177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42793
I20260924 23:08:47.229322 14305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42793 every 8 connection(s)
I20260924 23:08:47.229640 14177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:47.237200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14177
I20260924 23:08:47.237296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:08:47.237562 392 external_mini_cluster.cc:1428] Setting key abb8fa0ad132b86212ad1d7d1e98dc5f
I20260924 23:08:47.238931 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.230376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52587 (local address 127.0.98.62:37419)
0924 23:08:47.230421 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.230423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.230664 (+ 241us) server_negotiation.cc:407] Connection header received
0924 23:08:47.231197 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.231198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.231213 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.231258 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:47.232203 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.232718 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.233763 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.234026 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.234107 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.236190 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:47.236191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:47.236192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:47.236199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:47.237510 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.237976 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.237977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.237978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.238040 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.238285 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.238286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.238287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.238406 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:47.238483 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.238752 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.238844 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:47.239125 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.230360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:52587)
0924 23:08:47.230544 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.230581 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.231091 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.231326 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.231330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.231896 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:47.232102 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.232106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.232812 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.232813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.233654 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.233657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.234118 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.234119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.234268 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.234710 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:47.234718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:47.236086 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:47.237618 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:47.237625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:47.237627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:47.237863 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:47.238107 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:47.238107 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:47.238108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:47.238216 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:47.238473 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:47.238480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:47.238621 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.238730 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.238884 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260924 23:08:47.239900 14306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260924 23:08:47.239997 14306 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:47.240193 14306 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260924 23:08:47.240682 14115 ts_manager.cc:194] Registered new tserver with Master: a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793)
I20260924 23:08:47.241287 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52587
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:08:47.264411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:47.346328 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:47.346587 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:47.346611 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:47.347929 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:47.348002 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:47.348060 14311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:08:47.349639 14311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:47.349892 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:47.350226 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:47.355824 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.355829 14311 server_base.cc:1034] running on GCE node
W20260924 23:08:47.355811 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.355813 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.356320 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:47.356601 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:47.357782 14311 hybrid_clock.cc:648] HybridClock initialized: now 1790291327357782 us; error 43 us; skew 500 ppm
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:47.360559 14311 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:08:47.361079 14311 webserver.cc:460] Webserver started at http://127.0.98.2:33073/ using document root <none> and password file <none>
I20260924 23:08:47.361356 14311 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:47.361431 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:47.361593 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:47.362787 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad967a7e2628498cbf82c0eee2883da5"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "f2c6e56b8295c1a8f7d8c0bf5e370926"
server_key_iv: "d3c08f81d79ed64e7629ad8e9b54f382"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.363164 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad967a7e2628498cbf82c0eee2883da5"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "f2c6e56b8295c1a8f7d8c0bf5e370926"
server_key_iv: "d3c08f81d79ed64e7629ad8e9b54f382"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.364567 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:08:47.365339 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.365548 14311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:08:47.365661 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "ad967a7e2628498cbf82c0eee2883da5"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "f2c6e56b8295c1a8f7d8c0bf5e370926"
server_key_iv: "d3c08f81d79ed64e7629ad8e9b54f382"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.365741 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:47.389642 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:47.392498 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:47.392688 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:47.392956 14311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:47.393308 14311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:47.393369 14311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.393421 14311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:47.393471 14311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.400357 14311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42841
I20260924 23:08:47.400395 14439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42841 every 8 connection(s)
I20260924 23:08:47.400774 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:47.409430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14311
I20260924 23:08:47.409622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:08:47.409850 392 external_mini_cluster.cc:1428] Setting key d8eccf41a8bfeb82ddf2ea95741d230c
I20260924 23:08:47.411406 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.401733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51247 (local address 127.0.98.62:37419)
0924 23:08:47.401789 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.401792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.402110 (+ 318us) server_negotiation.cc:407] Connection header received
0924 23:08:47.402623 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.402630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.402643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.402705 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:47.403830 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.404369 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.405438 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.405630 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.405687 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.408032 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:47.408033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:47.408034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:47.408041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:47.409514 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.410250 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.410251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.410252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.410320 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.410642 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.410643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.410643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.410815 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:47.410870 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.411262 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.411341 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:47.411682 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.401741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:51247)
0924 23:08:47.401959 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.401998 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.402501 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.402765 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.402769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.403372 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:47.403702 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.403708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.404464 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.404465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.405346 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.405348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.405717 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.405718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.405838 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.406495 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:47.406503 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:47.407923 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:47.409740 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:47.409751 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:47.409754 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:47.410152 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:47.410389 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:47.410390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:47.410390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:47.410534 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:47.410886 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:47.410894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:47.411111 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.411225 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.411419 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260924 23:08:47.412199 14440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260924 23:08:47.412318 14440 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:47.412508 14440 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260924 23:08:47.412990 14115 ts_manager.cc:194] Registered new tserver with Master: ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841)
I20260924 23:08:47.413552 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51247
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:08:47.436632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:47.516892 14445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:47.517151 14445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:47.517175 14445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:47.518550 14445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:47.518594 14445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:47.518621 14445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:08:47.520181 14445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:47.520423 14445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:47.520740 14445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:47.526659 14451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.526805 14450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.526903 14453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.527072 14445 server_base.cc:1034] running on GCE node
I20260924 23:08:47.527269 14445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:47.527541 14445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:47.528716 14445 hybrid_clock.cc:648] HybridClock initialized: now 1790291327528692 us; error 32 us; skew 500 ppm
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:47.531935 14445 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:08:47.532653 14445 webserver.cc:460] Webserver started at http://127.0.98.3:41919/ using document root <none> and password file <none>
I20260924 23:08:47.532950 14445 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:47.533018 14445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:47.533162 14445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:08:47.535064 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8bb36e9b4294537b4aec27e5544488e"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8b9ccd5ce32f098858ca0741b625eede"
server_key_iv: "d6c86532976de6f07acdfed4030e9e0e"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.535519 14445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8bb36e9b4294537b4aec27e5544488e"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8b9ccd5ce32f098858ca0741b625eede"
server_key_iv: "d6c86532976de6f07acdfed4030e9e0e"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.537643 14445 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 23:08:47.538668 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.538841 14445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:47.538934 14445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "c8bb36e9b4294537b4aec27e5544488e"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8b9ccd5ce32f098858ca0741b625eede"
server_key_iv: "d6c86532976de6f07acdfed4030e9e0e"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.539014 14445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:47.557741 14445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:47.559757 14445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:47.559875 14445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:47.560071 14445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:47.560405 14445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:47.560438 14445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.560461 14445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:47.560511 14445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.566447 14445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39601
I20260924 23:08:47.566512 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39601 every 8 connection(s)
I20260924 23:08:47.566830 14445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:08:47.571758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14445
I20260924 23:08:47.571862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:08:47.572063 392 external_mini_cluster.cc:1428] Setting key a1b6e776c90523a272e02d6b9c0fc4f4
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:47.576642 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.567581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47519 (local address 127.0.98.62:37419)
0924 23:08:47.567632 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.567636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.567851 (+ 215us) server_negotiation.cc:407] Connection header received
0924 23:08:47.568333 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.568334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.568344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.568400 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:47.569620 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.570216 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.571300 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.571482 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.571539 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.573976 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:47.573978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:47.573978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:47.573985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:47.575409 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.575757 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.575757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.575758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.575817 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.576018 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.576019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.576019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.576117 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:47.576166 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.576446 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.576568 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:47.576915 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.567566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:47519)
0924 23:08:47.567729 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.567766 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.568252 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.568513 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.568517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.569291 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:47.569515 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.569521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.570312 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.570313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.571222 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.571225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.571615 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.571616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.571780 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.572401 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:47.572410 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:47.573853 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:47.575478 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:47.575484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:47.575485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:47.575694 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:47.575874 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:47.575875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:47.575875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:47.575967 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:47.576170 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:47.576175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:47.576288 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.576372 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.576493 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260924 23:08:47.577576 14574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260924 23:08:47.577700 14574 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:47.577917 14574 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260924 23:08:47.578361 14115 ts_manager.cc:194] Registered new tserver with Master: c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
I20260924 23:08:47.578768 14115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47519
I20260924 23:08:47.583843 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:08:47.591187 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.585222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.0.98.62:37419)
0924 23:08:47.585270 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.585272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.585420 (+ 148us) server_negotiation.cc:407] Connection header received
0924 23:08:47.585458 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.585459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.585468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.585519 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:47.586112 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.586582 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.587273 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.587447 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.587496 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.588637 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:47.588638 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:47.588638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:47.588644 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:47.590109 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.590428 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.590429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.590429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.590499 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:47.590691 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:47.590692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:47.590692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:47.590861 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:47.590942 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.591023 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.591086 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:08:47.592746 14115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:08:47.594488 14115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:08:47.601035 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.597136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.98.1:42793)
0924 23:08:47.597313 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.597317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.597393 (+ 76us) server_negotiation.cc:407] Connection header received
0924 23:08:47.597444 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.597446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.597461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.597539 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:47.597738 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.598868 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600120 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.600607 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600709 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.600770 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.600912 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.600952 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 23:08:47.601035 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.596892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.0.1:48470)
0924 23:08:47.597304 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.597316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.597385 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.597614 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.597615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.597617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:47.597653 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.597654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.598974 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.598975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.600025 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.600027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.600712 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.600791 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.600805 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.600897 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.600951 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":88,"threads_started":1}
I20260924 23:08:47.601135 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.596904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39601 (local address 127.0.0.1:50370)
0924 23:08:47.597462 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.597470 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.597511 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.597791 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.597792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.597795 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:47.597843 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.597844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.599098 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.599099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.600050 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.600053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.600940 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.600996 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.600999 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.601042 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.601077 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":79,"threads_started":1}
I20260924 23:08:47.601231 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.596856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42841 (local address 127.0.0.1:36798)
0924 23:08:47.597015 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.597056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.597120 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.597519 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.597522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.597525 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:47.597597 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.597599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.598647 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.598648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.599870 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.599872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.600854 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.600917 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.600921 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.600969 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.601009 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260924 23:08:47.601507 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.597137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.98.3:39601)
0924 23:08:47.597409 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.597414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.597581 (+ 167us) server_negotiation.cc:407] Connection header received
0924 23:08:47.597626 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.597628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.597645 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.597713 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:47.597953 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.598988 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600140 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.600806 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600862 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.600910 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.601384 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.601427 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260924 23:08:47.601507 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.596959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36798 (local address 127.0.98.2:42841)
0924 23:08:47.597348 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.597352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.597364 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:08:47.597408 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.597410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.597428 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.597515 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:47.597763 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.598523 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.599957 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.600689 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.600760 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.600817 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.601082 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.601425 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":85,"threads_started":1}
I20260924 23:08:47.602241 14374 tablet_service.cc:1467] Processing CreateTablet for tablet e3971284ef3c4cd7862cd423d199a8bf (DEFAULT_TABLE table=test-table [id=c2fd8638c4744352988cacf7b6b3833e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:47.602238 14240 tablet_service.cc:1467] Processing CreateTablet for tablet e3971284ef3c4cd7862cd423d199a8bf (DEFAULT_TABLE table=test-table [id=c2fd8638c4744352988cacf7b6b3833e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:47.602578 14374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3971284ef3c4cd7862cd423d199a8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:47.602579 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3971284ef3c4cd7862cd423d199a8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:47.605242 14591 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Bootstrap starting.
I20260924 23:08:47.605384 14592 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Bootstrap starting.
I20260924 23:08:47.605830 14508 tablet_service.cc:1467] Processing CreateTablet for tablet e3971284ef3c4cd7862cd423d199a8bf (DEFAULT_TABLE table=test-table [id=c2fd8638c4744352988cacf7b6b3833e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:47.606002 14591 tablet_bootstrap.cc:654] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:47.606127 14592 tablet_bootstrap.cc:654] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:47.606187 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3971284ef3c4cd7862cd423d199a8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:47.606604 14591 log.cc:824] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:47.606681 14592 log.cc:824] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:47.607353 14591 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: No bootstrap required, opened a new log
I20260924 23:08:47.607378 14592 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: No bootstrap required, opened a new log
I20260924 23:08:47.607436 14592 ts_tablet_manager.cc:1397] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:08:47.607479 14591 ts_tablet_manager.cc:1397] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:08:47.609136 14591 raft_consensus.cc:348] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.609314 14591 raft_consensus.cc:374] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:47.609462 14591 raft_consensus.cc:729] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204e31364a94edfb6e5c8b82de4927a, State: Initialized, Role: FOLLOWER
I20260924 23:08:47.609606 14595 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Bootstrap starting.
I20260924 23:08:47.609486 14592 raft_consensus.cc:348] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.609668 14592 raft_consensus.cc:374] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:47.609721 14592 raft_consensus.cc:729] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad967a7e2628498cbf82c0eee2883da5, State: Initialized, Role: FOLLOWER
I20260924 23:08:47.609768 14591 consensus_queue.cc:260] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.609840 14592 consensus_queue.cc:260] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.610183 14591 ts_tablet_manager.cc:1428] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 23:08:47.610181 14440 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
I20260924 23:08:47.610183 14592 ts_tablet_manager.cc:1428] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:08:47.610318 14306 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
I20260924 23:08:47.610348 14595 tablet_bootstrap.cc:654] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:47.610912 14595 log.cc:824] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:47.611637 14595 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: No bootstrap required, opened a new log
I20260924 23:08:47.611727 14595 ts_tablet_manager.cc:1397] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:08:47.613031 14595 raft_consensus.cc:348] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.613181 14595 raft_consensus.cc:374] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:47.613229 14595 raft_consensus.cc:729] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bb36e9b4294537b4aec27e5544488e, State: Initialized, Role: FOLLOWER
I20260924 23:08:47.613341 14595 consensus_queue.cc:260] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.613634 14595 ts_tablet_manager.cc:1428] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:08:47.613695 14574 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
W20260924 23:08:47.651523 14441 tablet.cc:2369] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:08:47.667259 14596 raft_consensus.cc:482] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:47.667447 14596 raft_consensus.cc:504] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.667883 14596 leader_election.cc:288] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601), a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793)
I20260924 23:08:47.671905 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.668215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35529 (local address 127.0.98.1:42793)
0924 23:08:47.668262 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.668264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.668276 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:08:47.668374 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.668376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.668389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.668457 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:47.668653 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.669442 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.670929 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.671497 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.671576 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.671587 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.671794 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.671835 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:08:47.671905 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.668245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60517 (local address 127.0.98.3:39601)
0924 23:08:47.668291 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.668292 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.668484 (+ 192us) server_negotiation.cc:407] Connection header received
0924 23:08:47.668511 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.668512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.668524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.668580 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:47.668779 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.669466 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.670929 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.671497 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.671576 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.671587 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.671794 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.671835 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:08:47.671914 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.668090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.98.2:35529)
0924 23:08:47.668179 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.668206 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.668272 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.668478 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.668479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.668482 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:47.668584 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.668586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.669542 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.669542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.670822 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.670825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.671608 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.671609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.671700 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.671706 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.671769 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.671814 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":16}
I20260924 23:08:47.671921 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.668090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39601 (local address 127.0.98.2:60517)
0924 23:08:47.668369 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.668380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:47.668446 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:47.668626 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:47.668626 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:47.668628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:47.668697 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.668698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.669542 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.669542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.670828 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:47.670830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.671608 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.671609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:47.671700 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.671707 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 23:08:47.671769 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 23:08:47.671814 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":124,"thread_start_us":139,"threads_started":1}
I20260924 23:08:47.672314 14260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3971284ef3c4cd7862cd423d199a8bf" candidate_uuid: "ad967a7e2628498cbf82c0eee2883da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204e31364a94edfb6e5c8b82de4927a" is_pre_election: true
I20260924 23:08:47.672435 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3971284ef3c4cd7862cd423d199a8bf" candidate_uuid: "ad967a7e2628498cbf82c0eee2883da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bb36e9b4294537b4aec27e5544488e" is_pre_election: true
I20260924 23:08:47.672492 14260 raft_consensus.cc:2454] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad967a7e2628498cbf82c0eee2883da5 in term 0.
I20260924 23:08:47.672610 14528 raft_consensus.cc:2454] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad967a7e2628498cbf82c0eee2883da5 in term 0.
I20260924 23:08:47.672724 14327 leader_election.cc:302] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a, ad967a7e2628498cbf82c0eee2883da5; no voters:
I20260924 23:08:47.672856 14596 raft_consensus.cc:2781] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:08:47.672948 14596 raft_consensus.cc:482] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:08:47.672982 14596 raft_consensus.cc:3037] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:47.673727 14596 raft_consensus.cc:504] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.673905 14596 leader_election.cc:288] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [CANDIDATE]: Term 1 election: Requested vote from peers c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601), a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793)
I20260924 23:08:47.674047 14260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3971284ef3c4cd7862cd423d199a8bf" candidate_uuid: "ad967a7e2628498cbf82c0eee2883da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a204e31364a94edfb6e5c8b82de4927a"
I20260924 23:08:47.674063 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3971284ef3c4cd7862cd423d199a8bf" candidate_uuid: "ad967a7e2628498cbf82c0eee2883da5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bb36e9b4294537b4aec27e5544488e"
I20260924 23:08:47.674129 14260 raft_consensus.cc:3037] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:47.674165 14528 raft_consensus.cc:3037] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:47.674878 14260 raft_consensus.cc:2454] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad967a7e2628498cbf82c0eee2883da5 in term 1.
I20260924 23:08:47.674901 14528 raft_consensus.cc:2454] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad967a7e2628498cbf82c0eee2883da5 in term 1.
I20260924 23:08:47.675041 14327 leader_election.cc:302] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a, ad967a7e2628498cbf82c0eee2883da5; no voters:
I20260924 23:08:47.675135 14596 raft_consensus.cc:2781] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:47.675295 14596 raft_consensus.cc:686] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 1 LEADER]: Becoming Leader. State: Replica: ad967a7e2628498cbf82c0eee2883da5, State: Running, Role: LEADER
I20260924 23:08:47.675396 14596 consensus_queue.cc:237] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:47.676244 14114 catalog_manager.cc:5495] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ad967a7e2628498cbf82c0eee2883da5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } health_report { overall_health: UNKNOWN } } }
I20260924 23:08:47.693214 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.690805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.0.98.2:42841)
0924 23:08:47.690869 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:47.690870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:47.690878 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:08:47.690955 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:47.690956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:47.690977 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:47.691040 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:08:47.691216 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.691678 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.692414 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:47.692598 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:47.692652 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:47.692789 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:08:47.692889 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:08:47.692948 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:47.693054 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 23:08:47.693071 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:08:47.695077 14260 raft_consensus.cc:1261] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Refusing update from remote peer ad967a7e2628498cbf82c0eee2883da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:47.695188 14528 raft_consensus.cc:1261] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Refusing update from remote peer ad967a7e2628498cbf82c0eee2883da5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:47.695466 14596 consensus_queue.cc:1035] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:47.695621 14601 consensus_queue.cc:1035] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:47.698683 14606 mvcc.cc:205] Tried to move back new op lower bound from 7333033278236364800 to 7333033278159126528. Current Snapshot: MvccSnapshot[applied={T|T < 7333033278236364800}]
I20260924 23:08:47.700145 14115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:08:47.700230 14115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:08:47.701082 14115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Sending DeleteTablet for 3 replicas of tablet e3971284ef3c4cd7862cd423d199a8bf
I20260924 23:08:47.701419 14374 tablet_service.cc:1514] Processing DeleteTablet for tablet e3971284ef3c4cd7862cd423d199a8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36798
I20260924 23:08:47.701439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14177
I20260924 23:08:47.701431 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet e3971284ef3c4cd7862cd423d199a8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50370
I20260924 23:08:47.701431 14240 tablet_service.cc:1514] Processing DeleteTablet for tablet e3971284ef3c4cd7862cd423d199a8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48470
I20260924 23:08:47.702955 14615 tablet_replica.cc:330] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: stopping tablet replica
I20260924 23:08:47.703058 14615 raft_consensus.cc:2229] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:08:47.703157 14615 raft_consensus.cc:2258] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:08:47.703308 14614 tablet_replica.cc:330] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: stopping tablet replica
I20260924 23:08:47.703392 14614 raft_consensus.cc:2229] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:08:47.703518 14614 raft_consensus.cc:2258] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:08:47.703934 14615 ts_tablet_manager.cc:1905] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:08:47.706084 14615 ts_tablet_manager.cc:1918] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:08:47.706147 14615 log.cc:1196] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/e3971284ef3c4cd7862cd423d199a8bf
I20260924 23:08:47.706323 14614 ts_tablet_manager.cc:1905] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:08:47.706408 14615 ts_tablet_manager.cc:1939] T e3971284ef3c4cd7862cd423d199a8bf P c8bb36e9b4294537b4aec27e5544488e: Deleting consensus metadata
I20260924 23:08:47.706835 14100 catalog_manager.cc:4841] TS c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601): tablet e3971284ef3c4cd7862cd423d199a8bf (table test-table [id=c2fd8638c4744352988cacf7b6b3833e]) successfully deleted
W20260924 23:08:47.707963 14099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42793 (error 108)
I20260924 23:08:47.708048 14614 ts_tablet_manager.cc:1918] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:08:47.708127 14614 log.cc:1196] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/e3971284ef3c4cd7862cd423d199a8bf
I20260924 23:08:47.708141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14311
I20260924 23:08:47.708386 14614 ts_tablet_manager.cc:1939] T e3971284ef3c4cd7862cd423d199a8bf P ad967a7e2628498cbf82c0eee2883da5: Deleting consensus metadata
I20260924 23:08:47.710780 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.710641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.0.1:48482)
0924 23:08:47.710704 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.710742 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 23:08:47.710978 14099 catalog_manager.cc:4584] TS a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3971284ef3c4cd7862cd423d199a8bf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42793: connect: Connection refused (error 111)
I20260924 23:08:47.713158 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:47.713039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42841 (local address 127.0.0.1:36826)
0924 23:08:47.713094 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:47.713127 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:08:47.713239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14445
I20260924 23:08:47.718225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14083
I20260924 23:08:47.723862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--webserver_interface=127.0.98.62
--webserver_port=37755
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:47.804765 14617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:47.804991 14617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:47.805014 14617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:47.806241 14617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:47.806284 14617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:47.806298 14617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:47.806310 14617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:47.806322 14617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:47.807792 14617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:47.808018 14617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:47.808199 14617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:47.816876 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.817037 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.817169 14617 server_base.cc:1034] running on GCE node
W20260924 23:08:47.816872 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.817517 14617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:47.817795 14617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:47.818989 14617 hybrid_clock.cc:648] HybridClock initialized: now 1790291327818995 us; error 45 us; skew 500 ppm
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:47.825632 14617 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:47.826164 14617 webserver.cc:460] Webserver started at http://127.0.98.62:37755/ using document root <none> and password file <none>
I20260924 23:08:47.826440 14617 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:47.826500 14617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:47.827944 14617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:47.828768 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.828986 14617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:47.829087 14617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "fdbe0481abb7416c876492d285e91656"
format_stamp: "Formatted at 2026-09-24 23:08:46 on dist-test-slave-ph9j"
server_key: "115036317581b2783b954798bfab0ba3"
server_key_iv: "0ee4c218b14d2c056bfa0a5605c9a04e"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.829520 14617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:47.856992 14617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:47.859517 14617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:47.859700 14617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:47.864033 14617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37419
I20260924 23:08:47.864073 14685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37419 every 8 connection(s)
I20260924 23:08:47.864506 14617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:47.866374 14686 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:08:47.867050 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap starting.
I20260924 23:08:47.869283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14617
I20260924 23:08:47.869611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42793
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37419
--webserver_port=42259
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:47.875996 14686 log.cc:824] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:47.878146 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:08:47.878444 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap complete.
I20260924 23:08:47.880730 14686 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.880949 14686 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbe0481abb7416c876492d285e91656, State: Initialized, Role: FOLLOWER
I20260924 23:08:47.881122 14686 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.881183 14686 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:08:47.881215 14686 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:08:47.881283 14686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:08:47.882351 14686 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.882459 14686 leader_election.cc:302] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdbe0481abb7416c876492d285e91656; no voters:
I20260924 23:08:47.882669 14686 leader_election.cc:288] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:08:47.882701 14690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:08:47.882812 14690 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 LEADER]: Becoming Leader. State: Replica: fdbe0481abb7416c876492d285e91656, State: Running, Role: LEADER
I20260924 23:08:47.882896 14690 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:47.882936 14686 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:47.883373 14691 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fdbe0481abb7416c876492d285e91656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260924 23:08:47.883479 14691 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:47.883786 14698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:47.884505 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdbe0481abb7416c876492d285e91656. Latest consensus state: current_term: 2 leader_uuid: "fdbe0481abb7416c876492d285e91656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260924 23:08:47.884601 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:47.884652 14698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:47.884791 14698 catalog_manager.cc:1259] Loaded cluster ID: e1253519c5d54a24b9671a6b74c2f608
I20260924 23:08:47.885041 14698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:08:47.885279 14698 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 23:08:47.888078 14698 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 23:08:47.889961 14698 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:08:47.890137 14698 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Loaded TSK: 0
I20260924 23:08:47.890355 14698 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:08:47.961869 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:47.962167 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:47.962205 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:47.963352 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:47.963423 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:47.963470 14688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:08:47.964999 14688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42259
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:47.965252 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:47.965480 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:47.971334 14716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.971329 14714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:47.971429 14713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:47.971690 14688 server_base.cc:1034] running on GCE node
I20260924 23:08:47.971933 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:47.972186 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:47.973340 14688 hybrid_clock.cc:648] HybridClock initialized: now 1790291327973311 us; error 44 us; skew 500 ppm
Sep 24 23:08:47 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:47.980191 14688 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:08:47.980741 14688 webserver.cc:460] Webserver started at http://127.0.98.1:42259/ using document root <none> and password file <none>
I20260924 23:08:47.980965 14688 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:47.981021 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:47.982499 14688 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:08:47.983215 14723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:47.983462 14688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:47.983577 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "a204e31364a94edfb6e5c8b82de4927a"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8192d020fb1892483887375734b2f675"
server_key_iv: "4756ed7ab72678d4d444955ac43f7c40"
server_key_version: "encryptionkey@0"
I20260924 23:08:47.984022 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:47.994418 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:47.996794 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:47.996954 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:47.997215 14688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:47.997717 14730 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:08:47.999198 14688 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:08:47.999258 14688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:08:47.999292 14688 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:08:48.000125 14688 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:08:48.000166 14688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:48.000249 14730 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Bootstrap starting.
I20260924 23:08:48.007248 14688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42793
I20260924 23:08:48.007725 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:08:48.010948 14730 log.cc:824] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:48.013484 14730 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:08:48.013804 14730 tablet_bootstrap.cc:492] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Bootstrap complete.
I20260924 23:08:48.014871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14688
I20260924 23:08:48.014994 14730 ts_tablet_manager.cc:1397] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20260924 23:08:48.015144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42841
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37419
--webserver_port=33073
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:48.016444 14730 raft_consensus.cc:348] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.016773 14730 raft_consensus.cc:729] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204e31364a94edfb6e5c8b82de4927a, State: Initialized, Role: FOLLOWER
I20260924 23:08:48.016921 14730 consensus_queue.cc:260] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.017206 14730 ts_tablet_manager.cc:1428] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260924 23:08:48.017318 14837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42793 every 8 connection(s)
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:48.022301 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.009016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60511 (local address 127.0.98.62:37419)
0924 23:08:48.009220 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.009230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.009280 (+ 50us) server_negotiation.cc:407] Connection header received
0924 23:08:48.010187 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.010195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.010253 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.010394 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:48.011301 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.012699 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.014241 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.014982 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.015423 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.018133 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:48.018138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:48.018138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:48.018150 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:48.020150 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.020682 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.020685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.020686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.020797 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.021141 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.021142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.021143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.021430 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:48.021535 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.021983 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.022125 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260924 23:08:48.022311 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.008640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:60511)
0924 23:08:48.008850 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.008902 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.009949 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.010465 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.010470 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.010864 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:48.011132 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.011141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.012833 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.012834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.014150 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.014153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.015151 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.015152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.015635 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.016265 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:48.016275 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:48.018009 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:48.020232 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.020239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.020242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.020575 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.020876 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.020877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.020877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.021033 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.021522 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:48.021529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:48.021716 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.021838 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.022057 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260924 23:08:48.023924 14838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260924 23:08:48.024032 14838 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:48.024232 14838 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260924 23:08:48.025103 14650 ts_manager.cc:194] Registered new tserver with Master: a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793)
I20260924 23:08:48.026273 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60511
I20260924 23:08:48.027215 14838 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 23:08:48.035974 14846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.026050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.1:42793)
0924 23:08:48.026230 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.026235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.026729 (+ 494us) server_negotiation.cc:407] Connection header received
0924 23:08:48.026878 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.026881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.026898 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.026962 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:48.027689 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.028736 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.029923 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.030363 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.030432 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.032785 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:48.032793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:48.032793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:48.032800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:48.034745 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.035188 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.035191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.035192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.035259 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.035451 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.035452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.035452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.035606 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:48.035668 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.035848 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.035910 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260924 23:08:48.036046 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.025775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.0.1:48498)
0924 23:08:48.026304 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.026432 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.026753 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.026992 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.026996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.027460 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:48.027531 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.027534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.028824 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.028825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.029816 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.029819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.030438 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.030440 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.030518 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.031058 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:48.031063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:48.032690 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:48.034837 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.034844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.034847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.035096 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.035328 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.035329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.035329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.035395 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.035669 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:48.035674 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:48.035735 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.035825 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.035940 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":92,"threads_started":1}
I20260924 23:08:48.036542 14756 tablet_service.cc:1514] Processing DeleteTablet for tablet e3971284ef3c4cd7862cd423d199a8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48498
I20260924 23:08:48.037055 14847 tablet_replica.cc:330] stopping tablet replica
I20260924 23:08:48.037163 14847 raft_consensus.cc:2229] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:08:48.037230 14847 raft_consensus.cc:2258] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:08:48.037534 14847 ts_tablet_manager.cc:1905] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:08:48.040079 14847 ts_tablet_manager.cc:1918] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:08:48.040174 14847 log.cc:1196] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/e3971284ef3c4cd7862cd423d199a8bf
I20260924 23:08:48.040441 14847 ts_tablet_manager.cc:1939] T e3971284ef3c4cd7862cd423d199a8bf P a204e31364a94edfb6e5c8b82de4927a: Deleting consensus metadata
I20260924 23:08:48.040992 14634 catalog_manager.cc:4841] TS a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793): tablet e3971284ef3c4cd7862cd423d199a8bf (table test-table [id=c2fd8638c4744352988cacf7b6b3833e]) successfully deleted
W20260924 23:08:48.103344 14843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:48.103559 14843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:48.103583 14843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:48.104763 14843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:48.104805 14843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:48.104862 14843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:08:48.106503 14843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33073
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:48.106760 14843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:48.106997 14843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:48.112028 14852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.112140 14855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.112028 14853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.112221 14843 server_base.cc:1034] running on GCE node
I20260924 23:08:48.112460 14843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:48.112656 14843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:48.113812 14843 hybrid_clock.cc:648] HybridClock initialized: now 1790291328113796 us; error 27 us; skew 500 ppm
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:48.117765 14843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:08:48.118422 14843 webserver.cc:460] Webserver started at http://127.0.98.2:33073/ using document root <none> and password file <none>
I20260924 23:08:48.118945 14843 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:48.119037 14843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:48.121279 14843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:08:48.122370 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.122570 14843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:48.122677 14843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "ad967a7e2628498cbf82c0eee2883da5"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "f2c6e56b8295c1a8f7d8c0bf5e370926"
server_key_iv: "d3c08f81d79ed64e7629ad8e9b54f382"
server_key_version: "encryptionkey@0"
I20260924 23:08:48.123231 14843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:48.135984 14843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:48.138973 14843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:48.139138 14843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:48.139428 14843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:48.139781 14843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:48.139820 14843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.139843 14843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:48.139892 14843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.147583 14843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42841
I20260924 23:08:48.147642 14975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42841 every 8 connection(s)
I20260924 23:08:48.148118 14843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:08:48.152055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14843
I20260924 23:08:48.152335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39601
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37419
--webserver_port=41919
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:48.164232 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.149020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56597 (local address 127.0.98.62:37419)
0924 23:08:48.149073 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.149076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.149362 (+ 286us) server_negotiation.cc:407] Connection header received
0924 23:08:48.150235 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.150237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.150250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.150313 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:48.151785 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.152574 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.154195 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.154485 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.154571 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.160575 (+ 6004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:48.160577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:48.160578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:48.160587 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:48.162494 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.163034 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.163035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.163036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.163102 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.163443 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.163444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.163444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.163590 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:48.163665 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.163985 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.164085 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:08:48.164355 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.149016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:56597)
0924 23:08:48.149214 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.149261 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.150124 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.150389 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.150393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.151329 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:48.151664 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.151670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.152687 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.152688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.154066 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.154069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.154583 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.154584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.154721 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.158368 (+ 3647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:48.158381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:48.160448 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:48.162579 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.162586 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.162588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.162924 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.163187 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.163188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.163189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.163348 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.163668 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:48.163674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:48.163840 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.163947 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.164109 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260924 23:08:48.164783 14976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260924 23:08:48.164873 14976 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:48.165057 14976 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260924 23:08:48.165432 14650 ts_manager.cc:194] Registered new tserver with Master: ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841)
I20260924 23:08:48.165829 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56597
W20260924 23:08:48.236726 14979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:48.236943 14979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:48.236966 14979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:48.238190 14979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:48.238232 14979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:48.238258 14979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:08:48.239732 14979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41919
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:48.239974 14979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:48.240209 14979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:48.245123 14984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.245285 14979 server_base.cc:1034] running on GCE node
W20260924 23:08:48.245157 14987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.245225 14985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.245589 14979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:48.245837 14979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:48.247030 14979 hybrid_clock.cc:648] HybridClock initialized: now 1790291328247004 us; error 42 us; skew 500 ppm
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:48.249360 14979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:08:48.249811 14979 webserver.cc:460] Webserver started at http://127.0.98.3:41919/ using document root <none> and password file <none>
I20260924 23:08:48.250124 14979 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:48.250216 14979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:48.251537 14979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:48.252107 14994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.252278 14979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:08:48.252389 14979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "c8bb36e9b4294537b4aec27e5544488e"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8b9ccd5ce32f098858ca0741b625eede"
server_key_iv: "d6c86532976de6f07acdfed4030e9e0e"
server_key_version: "encryptionkey@0"
I20260924 23:08:48.252743 14979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:48.266419 14979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:48.268325 14979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:48.268484 14979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:48.268720 14979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:48.269013 14979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:48.269070 14979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.269111 14979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:48.269140 14979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.275034 14979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39601
I20260924 23:08:48.275084 15107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39601 every 8 connection(s)
I20260924 23:08:48.275403 14979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:08:48.277681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 14979
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:08:48.285548 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.278124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.0.98.62:37419)
0924 23:08:48.278234 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.278236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.278244 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:08:48.278316 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.278317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.278329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.278386 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:48.279076 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.279510 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.280492 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.280765 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.280815 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.282047 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:48.282048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:48.282048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:48.282055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:48.284134 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.284559 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.284560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.284561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.284632 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.284855 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.284855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.284856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.285033 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:48.285201 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.285280 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.285373 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260924 23:08:48.286437 15110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.276087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:56987)
0924 23:08:48.276240 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.276274 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.276745 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.276986 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.276998 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.277552 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:48.277743 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.277748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.278608 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.278610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.279690 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.279692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.280029 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.280030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.280165 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.280754 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:48.280762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:48.282484 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:48.284444 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.284451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.284453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.284785 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.285083 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:48.285084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:48.285084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:48.285225 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:48.285546 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:48.285554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:48.285816 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.285977 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.286192 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 23:08:48.286707 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.276090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56987 (local address 127.0.98.62:37419)
0924 23:08:48.276151 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.276153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.276362 (+ 209us) server_negotiation.cc:407] Connection header received
0924 23:08:48.276841 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.276843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.276853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.276909 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:48.277912 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.278511 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.279773 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.279948 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.280000 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.282585 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:48.282586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:48.282586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:48.282600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:48.284376 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.284855 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.284855 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.284856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.284992 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:48.285300 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:48.285301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:48.285301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:48.285427 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:48.285512 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.286578 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.286650 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:08:48.287047 15108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37419
I20260924 23:08:48.287133 15108 heartbeater.cc:464] Registering TS with master...
I20260924 23:08:48.287278 15108 heartbeater.cc:511] Master 127.0.98.62:37419 requested a full tablet report, sending...
I20260924 23:08:48.287667 14650 ts_manager.cc:194] Registered new tserver with Master: c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
I20260924 23:08:48.288051 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56987
I20260924 23:08:48.297401 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:08:48.300218 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.297820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.0.98.62:37419)
0924 23:08:48.297942 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.297944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.297985 (+ 41us) server_negotiation.cc:407] Connection header received
0924 23:08:48.298057 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.298058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.298095 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.298166 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:08:48.298322 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.298840 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.299578 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.299759 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.299827 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.299959 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:08:48.300056 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:08:48.300118 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.300151 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.300166 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 23:08:48.300522 14650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:08:48.302243 14650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:08:48.304316 14756 tablet_service.cc:1467] Processing CreateTablet for tablet eee55659ac944c0caa638f9e8e22433d (DEFAULT_TABLE table=test-table [id=63fd91bd78ff468ca0d394d371218b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:48.304520 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee55659ac944c0caa638f9e8e22433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:48.306104 14730 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Bootstrap starting.
I20260924 23:08:48.306457 14730 tablet_bootstrap.cc:654] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:48.306991 14730 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: No bootstrap required, opened a new log
I20260924 23:08:48.307045 14730 ts_tablet_manager.cc:1397] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:48.307405 14730 raft_consensus.cc:348] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.307488 14730 raft_consensus.cc:374] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:48.307529 14730 raft_consensus.cc:729] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204e31364a94edfb6e5c8b82de4927a, State: Initialized, Role: FOLLOWER
I20260924 23:08:48.307580 14730 consensus_queue.cc:260] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.307663 14730 ts_tablet_manager.cc:1428] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:48.308153 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.304067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42841 (local address 127.0.0.1:36842)
0924 23:08:48.304119 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.304130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.304167 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.304589 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.304590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.304592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:48.304635 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.304636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.305689 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.305690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.306853 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.306856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.307887 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.307889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.307956 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.307977 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.308043 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.308084 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:08:48.308235 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.304166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.98.2:42841)
0924 23:08:48.304385 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.304389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.304398 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:08:48.304460 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.304462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.304479 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.304565 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:48.304695 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.305581 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.306944 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.307801 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.307881 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.307947 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.308081 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.308129 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
I20260924 23:08:48.308976 14910 tablet_service.cc:1467] Processing CreateTablet for tablet eee55659ac944c0caa638f9e8e22433d (DEFAULT_TABLE table=test-table [id=63fd91bd78ff468ca0d394d371218b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:48.309335 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee55659ac944c0caa638f9e8e22433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:48.309762 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.304072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39601 (local address 127.0.0.1:50384)
0924 23:08:48.304356 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.304365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.304409 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.306269 (+ 1860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.306270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.306272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:48.306313 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.306314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.307573 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.307574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.308697 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.308700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.309579 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.309580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.309630 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.309633 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.309678 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.309712 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":139,"thread_start_us":86,"threads_started":1}
I20260924 23:08:48.309830 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.304177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.0.98.3:39601)
0924 23:08:48.306010 (+ 1833us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.306014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.306022 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:08:48.306078 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.306080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.306095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.306171 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:48.306399 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.307433 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.308795 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.309491 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.309576 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.309631 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.309701 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.309743 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1801,"thread_start_us":100,"threads_started":1}
I20260924 23:08:48.310453 15042 tablet_service.cc:1467] Processing CreateTablet for tablet eee55659ac944c0caa638f9e8e22433d (DEFAULT_TABLE table=test-table [id=63fd91bd78ff468ca0d394d371218b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:08:48.310760 15042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee55659ac944c0caa638f9e8e22433d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:08:48.312304 15118 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Bootstrap starting.
I20260924 23:08:48.312958 15118 tablet_bootstrap.cc:654] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:48.313441 15119 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Bootstrap starting.
I20260924 23:08:48.313535 15118 log.cc:824] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:48.314174 15119 tablet_bootstrap.cc:654] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Neither blocks nor log segments found. Creating new log.
I20260924 23:08:48.314335 15118 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: No bootstrap required, opened a new log
I20260924 23:08:48.314409 15118 ts_tablet_manager.cc:1397] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:08:48.314745 15119 log.cc:824] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:48.315430 15119 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: No bootstrap required, opened a new log
I20260924 23:08:48.315515 15119 ts_tablet_manager.cc:1397] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:08:48.316258 15118 raft_consensus.cc:348] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.316395 15118 raft_consensus.cc:374] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:48.316418 15118 raft_consensus.cc:729] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad967a7e2628498cbf82c0eee2883da5, State: Initialized, Role: FOLLOWER
I20260924 23:08:48.316550 15118 consensus_queue.cc:260] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.316835 15118 ts_tablet_manager.cc:1428] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:08:48.316862 14976 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
I20260924 23:08:48.317150 15119 raft_consensus.cc:348] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.317323 15119 raft_consensus.cc:374] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:08:48.317351 15119 raft_consensus.cc:729] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8bb36e9b4294537b4aec27e5544488e, State: Initialized, Role: FOLLOWER
I20260924 23:08:48.317488 15119 consensus_queue.cc:260] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.317780 15119 ts_tablet_manager.cc:1428] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:08:48.317812 15108 heartbeater.cc:503] Master 127.0.98.62:37419 was elected leader, sending a full tablet report...
W20260924 23:08:48.398749 14977 tablet.cc:2369] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:08:48.518025 14839 tablet.cc:2369] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:08:48.526033 15109 tablet.cc:2369] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:08:48.574926 14844 raft_consensus.cc:482] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:48.575066 14844 raft_consensus.cc:504] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.575400 14844 leader_election.cc:288] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841), c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
I20260924 23:08:48.579427 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.575750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52515 (local address 127.0.98.3:39601)
0924 23:08:48.575806 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.575808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.575817 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:08:48.575898 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.575898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.575909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.575975 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:48.576162 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.576915 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.578475 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.579048 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.579126 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.579136 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.579315 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.579353 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:08:48.579427 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.575636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33591 (local address 127.0.98.2:42841)
0924 23:08:48.575683 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.575686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.575788 (+ 102us) server_negotiation.cc:407] Connection header received
0924 23:08:48.575894 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.575895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.575909 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.575975 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:08:48.576162 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.576907 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.578476 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.579048 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.579125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.579136 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.579313 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.579353 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:08:48.579464 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.575581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42841 (local address 127.0.98.1:33591)
0924 23:08:48.575719 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.575729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.575813 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.575996 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.575997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.576000 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:48.576091 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.576092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.576993 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.576994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.578385 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.578388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.579138 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.579139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.579207 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.579232 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.579293 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.579353 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":128,"thread_start_us":74,"threads_started":1}
I20260924 23:08:48.579471 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.575581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39601 (local address 127.0.98.1:52515)
0924 23:08:48.575712 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.575724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.575797 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.575997 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:48.575997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:48.576001 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:08:48.576087 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.576089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.576995 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.576996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.578386 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:48.578388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.579139 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.579140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:48.579213 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.579233 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 23:08:48.579293 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:08:48.579353 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":83,"thread_start_us":71,"threads_started":1}
I20260924 23:08:48.579840 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee55659ac944c0caa638f9e8e22433d" candidate_uuid: "a204e31364a94edfb6e5c8b82de4927a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad967a7e2628498cbf82c0eee2883da5" is_pre_election: true
I20260924 23:08:48.580026 14930 raft_consensus.cc:2454] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204e31364a94edfb6e5c8b82de4927a in term 0.
I20260924 23:08:48.580271 14727 leader_election.cc:302] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a, ad967a7e2628498cbf82c0eee2883da5; no voters:
I20260924 23:08:48.580257 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee55659ac944c0caa638f9e8e22433d" candidate_uuid: "a204e31364a94edfb6e5c8b82de4927a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bb36e9b4294537b4aec27e5544488e" is_pre_election: true
I20260924 23:08:48.580412 15062 raft_consensus.cc:2454] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a204e31364a94edfb6e5c8b82de4927a in term 0.
I20260924 23:08:48.580410 14844 raft_consensus.cc:2781] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:08:48.580477 14844 raft_consensus.cc:482] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:08:48.580513 14844 raft_consensus.cc:3037] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:48.581183 14844 raft_consensus.cc:504] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.581347 14844 leader_election.cc:288] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 1 election: Requested vote from peers ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841), c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
I20260924 23:08:48.581485 15062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee55659ac944c0caa638f9e8e22433d" candidate_uuid: "a204e31364a94edfb6e5c8b82de4927a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8bb36e9b4294537b4aec27e5544488e"
I20260924 23:08:48.581548 15062 raft_consensus.cc:3037] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:48.581553 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee55659ac944c0caa638f9e8e22433d" candidate_uuid: "a204e31364a94edfb6e5c8b82de4927a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad967a7e2628498cbf82c0eee2883da5"
I20260924 23:08:48.581630 14930 raft_consensus.cc:3037] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:08:48.582284 15062 raft_consensus.cc:2454] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204e31364a94edfb6e5c8b82de4927a in term 1.
I20260924 23:08:48.582472 14930 raft_consensus.cc:2454] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a204e31364a94edfb6e5c8b82de4927a in term 1.
I20260924 23:08:48.582468 14725 leader_election.cc:302] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a, c8bb36e9b4294537b4aec27e5544488e; no voters:
I20260924 23:08:48.582664 14844 raft_consensus.cc:2781] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:08:48.582798 14844 raft_consensus.cc:686] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 LEADER]: Becoming Leader. State: Replica: a204e31364a94edfb6e5c8b82de4927a, State: Running, Role: LEADER
I20260924 23:08:48.582873 14844 consensus_queue.cc:237] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.583704 14650 catalog_manager.cc:5495] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a reported cstate change: term changed from 0 to 1, leader changed from <none> to a204e31364a94edfb6e5c8b82de4927a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a204e31364a94edfb6e5c8b82de4927a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } health_report { overall_health: HEALTHY } } }
I20260924 23:08:48.593974 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.591599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.98.1:42793)
0924 23:08:48.591758 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:48.591759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:48.591766 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:08:48.591796 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:48.591797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:48.591812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:48.591879 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:08:48.592019 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.592475 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.593243 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:48.593424 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:48.593484 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:48.593646 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:08:48.593741 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:08:48.593792 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:48.593834 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 23:08:48.593849 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260924 23:08:48.595355 15062 raft_consensus.cc:1261] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Refusing update from remote peer a204e31364a94edfb6e5c8b82de4927a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:48.595387 14930 raft_consensus.cc:1261] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Refusing update from remote peer a204e31364a94edfb6e5c8b82de4927a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:08:48.595697 15126 consensus_queue.cc:1035] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:48.595865 14844 consensus_queue.cc:1035] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:08:48.598430 15132 mvcc.cc:205] Tried to move back new op lower bound from 7333033281924313088 to 7333033281876017152. Current Snapshot: MvccSnapshot[applied={T|T < 7333033281924313088}]
I20260924 23:08:48.600239 14650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:08:48.600314 14650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:08:48.600930 14650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Sending DeleteTablet for 3 replicas of tablet eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:48.601215 14910 tablet_service.cc:1514] Processing DeleteTablet for tablet eee55659ac944c0caa638f9e8e22433d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36842
I20260924 23:08:48.601311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14688
I20260924 23:08:48.601285 15042 tablet_service.cc:1514] Processing DeleteTablet for tablet eee55659ac944c0caa638f9e8e22433d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:08:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50384
I20260924 23:08:48.602210 15141 tablet_replica.cc:330] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: stopping tablet replica
I20260924 23:08:48.602314 15141 raft_consensus.cc:2229] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:08:48.602407 15141 raft_consensus.cc:2258] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:08:48.602624 15140 tablet_replica.cc:330] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: stopping tablet replica
I20260924 23:08:48.602707 15140 raft_consensus.cc:2229] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:08:48.602797 15140 raft_consensus.cc:2258] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:08:48.603029 15141 ts_tablet_manager.cc:1905] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:08:48.603225 15140 ts_tablet_manager.cc:1905] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:08:48.604722 15141 ts_tablet_manager.cc:1918] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:08:48.604809 15141 log.cc:1196] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:48.605072 15141 ts_tablet_manager.cc:1939] T eee55659ac944c0caa638f9e8e22433d P c8bb36e9b4294537b4aec27e5544488e: Deleting consensus metadata
I20260924 23:08:48.605186 15140 ts_tablet_manager.cc:1918] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:08:48.605285 15140 log.cc:1196] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:48.605569 15140 ts_tablet_manager.cc:1939] T eee55659ac944c0caa638f9e8e22433d P ad967a7e2628498cbf82c0eee2883da5: Deleting consensus metadata
I20260924 23:08:48.605623 14635 catalog_manager.cc:4841] TS c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601): tablet eee55659ac944c0caa638f9e8e22433d (table test-table [id=63fd91bd78ff468ca0d394d371218b77]) successfully deleted
I20260924 23:08:48.606061 14637 catalog_manager.cc:4841] TS ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841): tablet eee55659ac944c0caa638f9e8e22433d (table test-table [id=63fd91bd78ff468ca0d394d371218b77]) successfully deleted
W20260924 23:08:48.607155 14634 connection.cc:535] client connection to 127.0.98.1:42793 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 23:08:48.607275 14634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:08:48.607488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14843
I20260924 23:08:48.608073 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.607625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42793 (local address 127.0.0.1:48514)
0924 23:08:48.608001 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.608037 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350}
W20260924 23:08:48.608912 14634 catalog_manager.cc:4584] TS a204e31364a94edfb6e5c8b82de4927a (127.0.98.1:42793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eee55659ac944c0caa638f9e8e22433d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42793: connect: Connection refused (error 111)
I20260924 23:08:48.612390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14979
I20260924 23:08:48.616888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 14617
I20260924 23:08:48.622394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--webserver_interface=127.0.98.62
--webserver_port=37755
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:48.703022 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:48.703234 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:48.703258 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:48.704452 15143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:08:48.704496 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:48.704511 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:48.704524 15143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:08:48.704593 15143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:08:48.706188 15143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37755
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:48.706465 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:48.706720 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:48.711669 15151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.711679 15148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.711828 15143 server_base.cc:1034] running on GCE node
W20260924 23:08:48.711679 15149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.712186 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:48.712476 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:48.713652 15143 hybrid_clock.cc:648] HybridClock initialized: now 1790291328713628 us; error 36 us; skew 500 ppm
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:48.716086 15143 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:08:48.716508 15143 webserver.cc:460] Webserver started at http://127.0.98.62:37755/ using document root <none> and password file <none>
I20260924 23:08:48.716814 15143 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:48.716859 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:48.718266 15143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:08:48.719007 15158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.719197 15143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:48.719277 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "fdbe0481abb7416c876492d285e91656"
format_stamp: "Formatted at 2026-09-24 23:08:46 on dist-test-slave-ph9j"
server_key: "115036317581b2783b954798bfab0ba3"
server_key_iv: "0ee4c218b14d2c056bfa0a5605c9a04e"
server_key_version: "encryptionkey@0"
I20260924 23:08:48.719604 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:48.735368 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:48.737664 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:48.737775 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:48.741741 15143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37419
I20260924 23:08:48.741806 15210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37419 every 8 connection(s)
I20260924 23:08:48.742200 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:08:48.743777 15211 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:08:48.744376 15211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap starting.
I20260924 23:08:48.747169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15143
I20260924 23:08:48.747494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42793
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37419
--webserver_port=42259
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:48.753139 15211 log.cc:824] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:48.755439 15211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:08:48.755759 15211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656: Bootstrap complete.
I20260924 23:08:48.758074 15211 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:48.758302 15211 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdbe0481abb7416c876492d285e91656, State: Initialized, Role: FOLLOWER
I20260924 23:08:48.758471 15211 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:48.758530 15211 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:08:48.758580 15211 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:08:48.758630 15211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 2 FOLLOWER]: Advancing to term 3
I20260924 23:08:48.759665 15211 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:48.759780 15211 leader_election.cc:302] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdbe0481abb7416c876492d285e91656; no voters:
I20260924 23:08:48.759991 15211 leader_election.cc:288] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 23:08:48.760025 15215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 23:08:48.760135 15215 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [term 3 LEADER]: Becoming Leader. State: Replica: fdbe0481abb7416c876492d285e91656, State: Running, Role: LEADER
I20260924 23:08:48.760221 15215 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260924 23:08:48.760263 15211 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:08:48.760593 15216 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fdbe0481abb7416c876492d285e91656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260924 23:08:48.760674 15216 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:48.761133 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdbe0481abb7416c876492d285e91656. Latest consensus state: current_term: 3 leader_uuid: "fdbe0481abb7416c876492d285e91656" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdbe0481abb7416c876492d285e91656" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260924 23:08:48.761216 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdbe0481abb7416c876492d285e91656 [sys.catalog]: This master's current role is: LEADER
I20260924 23:08:48.761338 15221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:08:48.762168 15221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:08:48.762269 15221 catalog_manager.cc:1259] Loaded cluster ID: e1253519c5d54a24b9671a6b74c2f608
I20260924 23:08:48.762303 15221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 23:08:48.762509 15221 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f23039739ad google::LogMessage::Fail() at ??:0
@ 0x7f230397688a google::LogMessage::SendToLog() at ??:0
@ 0x7f23039734de google::LogMessage::Flush() at ??:0
@ 0x7f2303974a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f230b8940f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f230b89d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f23058410d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2305836954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f230669a6db start_thread at ??:0
@ 0x7f230247161f clone at ??:0
W20260924 23:08:48.837633 15213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:48.837860 15213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:48.837931 15213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:48.839087 15213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:48.839150 15213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:48.839195 15213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:08:48.840631 15213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42259
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:48.840888 15213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:48.841114 15213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:48.846176 15247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.846292 15213 server_base.cc:1034] running on GCE node
W20260924 23:08:48.846185 15248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.846189 15250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.846777 15213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:48.847080 15213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:48.848245 15213 hybrid_clock.cc:648] HybridClock initialized: now 1790291328848233 us; error 30 us; skew 500 ppm
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:48.851089 15213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:08:48.851619 15213 webserver.cc:460] Webserver started at http://127.0.98.1:42259/ using document root <none> and password file <none>
I20260924 23:08:48.851886 15213 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:48.851955 15213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:48.853747 15213 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:08:48.854552 15257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.854792 15213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:08:48.854888 15213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "a204e31364a94edfb6e5c8b82de4927a"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8192d020fb1892483887375734b2f675"
server_key_iv: "4756ed7ab72678d4d444955ac43f7c40"
server_key_version: "encryptionkey@0"
I20260924 23:08:48.855396 15213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:48.872763 15213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:48.875106 15213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:48.875252 15213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:48.875509 15213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:48.876019 15264 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:08:48.877974 15213 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:08:48.878042 15213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:08:48.878088 15213 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:08:48.878700 15213 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:08:48.878755 15213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:08:48.878823 15264 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Bootstrap starting.
I20260924 23:08:48.886264 15213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42793
I20260924 23:08:48.886667 15213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:08:48.889755 15264 log.cc:824] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:08:48.890126 15371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42793 every 8 connection(s)
I20260924 23:08:48.892336 15264 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:08:48.892642 15264 tablet_bootstrap.cc:492] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Bootstrap complete.
I20260924 23:08:48.892988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15213
I20260924 23:08:48.893229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42841
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37419
--webserver_port=33073
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:08:48.893860 15264 ts_tablet_manager.cc:1397] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260924 23:08:48.895427 15264 raft_consensus.cc:348] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.895758 15264 raft_consensus.cc:729] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a204e31364a94edfb6e5c8b82de4927a, State: Initialized, Role: FOLLOWER
I20260924 23:08:48.895929 15264 consensus_queue.cc:260] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:48.896214 15264 ts_tablet_manager.cc:1428] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260924 23:08:48.961151 15374 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:08:48.887310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:54651)
0924 23:08:48.887483 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.887525 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:48.887967 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:48.960943 (+ 72976us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
W20260924 23:08:48.961439 15372 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37419 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37419: Client connection negotiation failed: client connection to 127.0.98.62:37419: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:08:48.961808 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.961667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:50631)
0924 23:08:48.961732 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.961781 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 23:08:48.962217 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:48.962123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:33563)
0924 23:08:48.962169 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:48.962193 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 23:08:48.962360 15372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:08:48.974326 15376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:48.974522 15376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:48.974553 15376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:48.975689 15376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:48.975730 15376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:48.975755 15376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:08:48.977180 15376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33073
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:48.977416 15376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:48.977646 15376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:48.982580 15382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.982648 15385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:48.982599 15383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:48.982784 15376 server_base.cc:1034] running on GCE node
I20260924 23:08:48.982975 15376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:48.983161 15376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:48.984292 15376 hybrid_clock.cc:648] HybridClock initialized: now 1790291328984282 us; error 27 us; skew 500 ppm
Sep 24 23:08:48 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:48.986881 15376 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:08:48.987316 15376 webserver.cc:460] Webserver started at http://127.0.98.2:33073/ using document root <none> and password file <none>
I20260924 23:08:48.987542 15376 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:48.987613 15376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:48.988974 15376 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:48.989573 15392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:48.989733 15376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:08:48.989832 15376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "ad967a7e2628498cbf82c0eee2883da5"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "f2c6e56b8295c1a8f7d8c0bf5e370926"
server_key_iv: "d3c08f81d79ed64e7629ad8e9b54f382"
server_key_version: "encryptionkey@0"
I20260924 23:08:48.990254 15376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:49.012269 15376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:49.014709 15376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:49.014873 15376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:49.015123 15376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:49.015424 15376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:49.015478 15376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:49.015523 15376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:49.015563 15376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:49.021380 15376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42841
I20260924 23:08:49.021440 15505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42841 every 8 connection(s)
I20260924 23:08:49.021773 15376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:08:49.022841 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.022479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:58873)
0924 23:08:49.022629 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.022700 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
W20260924 23:08:49.023382 15506 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37419 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37419: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
I20260924 23:08:49.023671 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.023578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:35471)
0924 23:08:49.023625 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.023653 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 23:08:49.023936 15508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.023880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:59699)
0924 23:08:49.023902 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.023922 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 23:08:49.024049 15506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:08:49.028750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15376
I20260924 23:08:49.029023 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39601
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37419
--webserver_port=41919
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:08:49.109597 15509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:08:49.109802 15509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:08:49.109825 15509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:08:49.110996 15509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:08:49.111040 15509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:08:49.111068 15509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:08:49.112421 15509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41919
--tserver_master_addrs=127.0.98.62:37419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:08:49.112612 15509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:08:49.112797 15509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:08:49.117643 15517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:49.117766 15515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:08:49.117643 15514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:08:49.117854 15509 server_base.cc:1034] running on GCE node
I20260924 23:08:49.118162 15509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:08:49.118404 15509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:08:49.119545 15509 hybrid_clock.cc:648] HybridClock initialized: now 1790291329119532 us; error 32 us; skew 500 ppm
Sep 24 23:08:49 dist-test-slave-ph9j krb5kdc[14070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:08:49.122076 15509 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:08:49.122511 15509 webserver.cc:460] Webserver started at http://127.0.98.3:41919/ using document root <none> and password file <none>
I20260924 23:08:49.122759 15509 fs_manager.cc:359] Metadata directory not provided
I20260924 23:08:49.122808 15509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:08:49.124274 15509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:08:49.124926 15524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:49.125130 15509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:08:49.125234 15509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "c8bb36e9b4294537b4aec27e5544488e"
format_stamp: "Formatted at 2026-09-24 23:08:47 on dist-test-slave-ph9j"
server_key: "8b9ccd5ce32f098858ca0741b625eede"
server_key_iv: "d6c86532976de6f07acdfed4030e9e0e"
server_key_version: "encryptionkey@0"
I20260924 23:08:49.125602 15509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:08:49.146371 15509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:08:49.148484 15509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:08:49.148658 15509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:08:49.148916 15509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:08:49.149233 15509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:08:49.149290 15509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:49.149331 15509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:08:49.149362 15509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:08:49.155293 15509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39601
I20260924 23:08:49.155341 15637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39601 every 8 connection(s)
I20260924 23:08:49.155673 15509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:08:49.156715 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.156328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:40235)
0924 23:08:49.156481 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.156574 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260924 23:08:49.157114 15638 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37419 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37419: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
I20260924 23:08:49.157444 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.157330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:32893)
0924 23:08:49.157388 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.157426 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:08:49.157787 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.157733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:35759)
0924 23:08:49.157758 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.157776 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 23:08:49.157927 15638 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:08:49.163718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15509
I20260924 23:08:49.962929 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:49.962716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:45657)
0924 23:08:49.962835 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:49.962870 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 23:08:50.024552 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.024359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:35199)
0924 23:08:50.024474 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:50.024510 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260924 23:08:50.148527 15644 raft_consensus.cc:482] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:50.148725 15644 raft_consensus.cc:504] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:50.149070 15644 leader_election.cc:288] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841), c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
Sep 24 23:08:50 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 23:08:50 dist-test-slave-ph9j krb5kdc[14070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 23:08:50.158434 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.158226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:44715)
0924 23:08:50.158335 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:50.158372 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260924 23:08:50.160094 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.149444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46587 (local address 127.0.98.3:39601)
0924 23:08:50.149599 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:50.149612 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:50.149637 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:08:50.150166 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:50.150169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:50.150239 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:50.150383 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:50.151114 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.152194 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.153118 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.153525 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.153661 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:50.156188 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:50.156198 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:50.156199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:50.156209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:50.158270 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:50.158760 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:50.158763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:50.158763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:50.158845 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:50.159103 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:50.159104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:50.159104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:50.159446 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:50.159532 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:50.159838 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 23:08:50.159979 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":52,"threads_started":1}
I20260924 23:08:50.160101 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.149299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39601 (local address 127.0.98.1:46587)
0924 23:08:50.149483 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:50.149493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:50.149603 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:50.150392 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:50.150395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:50.150815 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:50.150991 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:50.150997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.152287 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.152288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:50.153025 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:50.153028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.153625 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.153626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:50.153732 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:50.154361 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:50.154369 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:50.156070 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:50.158354 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:50.158361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:50.158363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:50.158683 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:50.158897 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:50.158898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:50.158898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:50.159029 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:50.159510 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:50.159517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:50.159634 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 23:08:50.159757 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 23:08:50.159969 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":114,"thread_start_us":65,"threads_started":1}
I20260924 23:08:50.160135 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.149450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41013 (local address 127.0.98.2:42841)
0924 23:08:50.149638 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 23:08:50.149646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:08:50.149672 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:08:50.150166 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:08:50.150168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:08:50.150212 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:08:50.150351 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:08:50.151129 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.152142 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.153115 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.153518 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.153646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:50.156632 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:08:50.156636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:08:50.156636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:08:50.156647 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:08:50.158531 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:50.158956 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:50.158958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:50.158958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:50.159039 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:08:50.159256 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:08:50.159257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:08:50.159257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:08:50.159660 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:08:50.159722 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:08:50.159902 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 23:08:50.160024 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260924 23:08:50.160784 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.149289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42841 (local address 127.0.98.1:41013)
0924 23:08:50.149392 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:50.149407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:08:50.149481 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:08:50.150391 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:08:50.150394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:08:50.150811 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:08:50.150991 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:50.150997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.152240 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.152241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:50.153022 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:08:50.153024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:08:50.153621 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:08:50.153622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:08:50.153733 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:08:50.154417 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:08:50.154428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:08:50.156524 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:08:50.158628 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:50.158629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:50.158629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:50.158884 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:50.159140 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:08:50.159140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:08:50.159141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:08:50.159191 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:08:50.159733 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:08:50.159734 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:08:50.159772 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 23:08:50.159840 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:08:50.160025 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":111}
W20260924 23:08:50.160871 15259 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
W20260924 23:08:50.161475 15261 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:50.161535 15261 leader_election.cc:302] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a; no voters: ad967a7e2628498cbf82c0eee2883da5, c8bb36e9b4294537b4aec27e5544488e
I20260924 23:08:50.161641 15644 raft_consensus.cc:2726] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:08:50.963588 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:50.963397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:34519)
0924 23:08:50.963508 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:50.963540 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 23:08:51.025215 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:51.025006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:60179)
0924 23:08:51.025143 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:51.025176 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":59,"threads_started":1}
I20260924 23:08:51.159068 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:51.158828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:54027)
0924 23:08:51.158961 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:51.159002 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 23:08:51.964198 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:51.963986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:51653)
0924 23:08:51.964119 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:51.964153 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260924 23:08:52.025826 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:52.025647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:47061)
0924 23:08:52.025754 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:52.025790 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260924 23:08:52.159834 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:52.159580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:53003)
0924 23:08:52.159709 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:52.159784 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260924 23:08:52.256155 15655 raft_consensus.cc:482] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:52.256239 15655 raft_consensus.cc:504] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:52.256392 15655 leader_election.cc:288] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841), c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
W20260924 23:08:52.256783 15259 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
W20260924 23:08:52.256902 15261 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:52.256980 15261 leader_election.cc:302] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a; no voters: ad967a7e2628498cbf82c0eee2883da5, c8bb36e9b4294537b4aec27e5544488e
I20260924 23:08:52.257068 15655 raft_consensus.cc:2726] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:08:52.964812 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:52.964622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:41693)
0924 23:08:52.964734 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:52.964768 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260924 23:08:53.026409 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:53.026213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:41621)
0924 23:08:53.026330 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:53.026368 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260924 23:08:53.160562 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:53.160350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:52285)
0924 23:08:53.160473 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:53.160516 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260924 23:08:53.965585 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:53.965290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:47251)
0924 23:08:53.965452 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:53.965487 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260924 23:08:54.019343 15660 raft_consensus.cc:482] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:54.019459 15660 raft_consensus.cc:504] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:54.019647 15660 leader_election.cc:288] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841), c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
W20260924 23:08:54.020044 15259 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
W20260924 23:08:54.020123 15261 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:54.020205 15261 leader_election.cc:302] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a; no voters: ad967a7e2628498cbf82c0eee2883da5, c8bb36e9b4294537b4aec27e5544488e
I20260924 23:08:54.020274 15660 raft_consensus.cc:2726] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:08:54.027047 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:54.026842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:34407)
0924 23:08:54.026973 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:54.027010 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260924 23:08:54.161216 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:54.160993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:47423)
0924 23:08:54.161125 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:54.161166 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 23:08:54.966308 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:54.966097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:55679)
0924 23:08:54.966219 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:54.966256 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 23:08:55.027603 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:55.027420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:49929)
0924 23:08:55.027533 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:55.027567 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260924 23:08:55.161971 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:55.161688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:36795)
0924 23:08:55.161827 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:55.161865 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260924 23:08:55.967020 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:55.966787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:55823)
0924 23:08:55.966938 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:55.966973 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260924 23:08:56.028358 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:56.028111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:34165)
0924 23:08:56.028257 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:56.028308 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260924 23:08:56.162623 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:56.162412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:47097)
0924 23:08:56.162539 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:56.162579 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260924 23:08:56.967684 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:56.967467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:53933)
0924 23:08:56.967602 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:56.967637 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260924 23:08:56.993949 15670 raft_consensus.cc:482] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:08:56.994058 15670 raft_consensus.cc:504] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad967a7e2628498cbf82c0eee2883da5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42841 } } peers { permanent_uuid: "c8bb36e9b4294537b4aec27e5544488e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39601 } } peers { permanent_uuid: "a204e31364a94edfb6e5c8b82de4927a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42793 } }
I20260924 23:08:56.994249 15670 leader_election.cc:288] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841), c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601)
W20260924 23:08:56.994614 15261 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad967a7e2628498cbf82c0eee2883da5 (127.0.98.2:42841): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
W20260924 23:08:56.994710 15259 leader_election.cc:341] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8bb36e9b4294537b4aec27e5544488e (127.0.98.3:39601): Not found: Tablet not found: eee55659ac944c0caa638f9e8e22433d
I20260924 23:08:56.994796 15259 leader_election.cc:302] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a204e31364a94edfb6e5c8b82de4927a; no voters: ad967a7e2628498cbf82c0eee2883da5, c8bb36e9b4294537b4aec27e5544488e
I20260924 23:08:56.994895 15670 raft_consensus.cc:2726] T eee55659ac944c0caa638f9e8e22433d P a204e31364a94edfb6e5c8b82de4927a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:08:57.029029 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:57.028766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:42393)
0924 23:08:57.028941 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:57.028978 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":118,"threads_started":1}
I20260924 23:08:57.163290 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:57.163075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:37227)
0924 23:08:57.163201 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:57.163233 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260924 23:08:57.968355 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:57.968133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:38667)
0924 23:08:57.968267 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:57.968301 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260924 23:08:58.029667 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:58.029476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:51725)
0924 23:08:58.029589 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:58.029624 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 23:08:58.163924 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:58.163713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:54311)
0924 23:08:58.163847 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:58.163881 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260924 23:08:58.969015 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:58.968797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:53643)
0924 23:08:58.968930 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:58.968969 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":63,"threads_started":1}
I20260924 23:08:59.030362 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:59.030161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:60221)
0924 23:08:59.030280 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:59.030319 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 23:08:59.164559 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:59.164358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:58551)
0924 23:08:59.164478 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:59.164514 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260924 23:08:59.969689 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:08:59.969440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:44887)
0924 23:08:59.969605 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:08:59.969641 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
I20260924 23:09:00.031055 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:00.030824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:60917)
0924 23:09:00.030959 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:00.030993 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260924 23:09:00.165309 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:00.165084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:58563)
0924 23:09:00.165221 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:00.165263 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 23:09:00.970470 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:00.970263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:56631)
0924 23:09:00.970384 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:00.970424 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260924 23:09:01.031704 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:01.031492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:35635)
0924 23:09:01.031629 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:01.031663 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
I20260924 23:09:01.165978 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:01.165753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:35969)
0924 23:09:01.165902 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:01.165935 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260924 23:09:01.971093 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:01.970874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:38017)
0924 23:09:01.971012 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:01.971048 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260924 23:09:02.032447 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:02.032190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:54481)
0924 23:09:02.032365 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:02.032403 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260924 23:09:02.166612 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:02.166406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:52623)
0924 23:09:02.166527 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:02.166564 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260924 23:09:02.971839 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:02.971574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:38701)
0924 23:09:02.971752 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:02.971792 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260924 23:09:03.033275 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:03.033034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:43591)
0924 23:09:03.033173 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:03.033230 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":56,"threads_started":1}
I20260924 23:09:03.167330 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:03.167118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:55853)
0924 23:09:03.167246 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:03.167284 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 23:09:03.972594 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:03.972386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.1:57943)
0924 23:09:03.972510 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:03.972546 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260924 23:09:04.034189 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:04.033975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.2:46907)
0924 23:09:04.034105 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:04.034145 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260924 23:09:04.168075 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:04.167848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37419 (local address 127.0.98.3:38125)
0924 23:09:04.167996 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:04.168031 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260924 23:09:04.172148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 15213
I20260924 23:09:04.176895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 15376
I20260924 23:09:04.181162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 15509
2026-09-24T23:09:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18450 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:09:04 dist-test-slave-ph9j krb5kdc[15696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:09:04 dist-test-slave-ph9j krb5kdc[15696](info): set up 2 sockets
Sep 24 23:09:04 dist-test-slave-ph9j krb5kdc[15696](info): commencing operation
krb5kdc: starting...
W20260924 23:09:05.226040 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:09:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:09:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:09:05.345172 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:05.424674 15709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:05.424937 15709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:05.424973 15709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:05.426133 15709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 23:09:05.426203 15709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 23:09:05.426224 15709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 23:09:05.426244 15709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:09:05.426276 15709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:05.426296 15709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:05.426316 15709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:09:05.426338 15709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:09:05.427814 15709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:05.428102 15709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:05.428328 15709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:09:05.433324 15709 server_base.cc:1034] running on GCE node
W20260924 23:09:05.433481 15715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:05.433486 15714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:05.433523 15717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:05.433862 15709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:05.434223 15709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:05.435402 15709 hybrid_clock.cc:648] HybridClock initialized: now 1790291345435376 us; error 47 us; skew 500 ppm
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:05.438072 15709 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:09:05.438552 15709 webserver.cc:460] Webserver started at http://127.0.98.62:41019/ using document root <none> and password file <none>
I20260924 23:09:05.438803 15709 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:05.438872 15709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:05.439031 15709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:09:05.440181 15709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "08589a56604d46bd977af36140d0f0e9"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "3a50a11a14712670a082dc8aa3738218"
server_key_iv: "736d9b0889edac30bc19156213df902c"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.440543 15709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "08589a56604d46bd977af36140d0f0e9"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "3a50a11a14712670a082dc8aa3738218"
server_key_iv: "736d9b0889edac30bc19156213df902c"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.441956 15709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 23:09:05.442720 15724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.442902 15709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:05.443003 15709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "08589a56604d46bd977af36140d0f0e9"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "3a50a11a14712670a082dc8aa3738218"
server_key_iv: "736d9b0889edac30bc19156213df902c"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.443084 15709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:05.468230 15709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:05.470381 15709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:05.470594 15709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:05.474745 15709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43791
I20260924 23:09:05.474795 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43791 every 8 connection(s)
I20260924 23:09:05.475178 15709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:09:05.476037 15777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:05.478555 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap starting.
I20260924 23:09:05.479166 15777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:05.479472 15777 log.cc:824] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:05.479892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15709
I20260924 23:09:05.479990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:09:05.480181 392 external_mini_cluster.cc:1428] Setting key 107a8b303e5b0c5a8aa8f6a08959a832
I20260924 23:09:05.480311 15777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: No bootstrap required, opened a new log
I20260924 23:09:05.482602 15777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:05.482780 15777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:05.482824 15777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08589a56604d46bd977af36140d0f0e9, State: Initialized, Role: FOLLOWER
I20260924 23:09:05.482966 15777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:05.483063 15777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:09:05.483098 15777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:09:05.483146 15777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:05.483851 15777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:05.483963 15777 leader_election.cc:302] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08589a56604d46bd977af36140d0f0e9; no voters:
I20260924 23:09:05.484109 15777 leader_election.cc:288] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:09:05.484162 15782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:09:05.484344 15777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:09:05.484360 15782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 LEADER]: Becoming Leader. State: Replica: 08589a56604d46bd977af36140d0f0e9, State: Running, Role: LEADER
I20260924 23:09:05.485596 15782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:05.486240 15784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08589a56604d46bd977af36140d0f0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } } }
I20260924 23:09:05.486510 15784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: This master's current role is: LEADER
W20260924 23:09:05.486603 15796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:09:05.486646 15796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:09:05.486408 15783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08589a56604d46bd977af36140d0f0e9. Latest consensus state: current_term: 1 leader_uuid: "08589a56604d46bd977af36140d0f0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } } }
I20260924 23:09:05.486939 15783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:09:05.487043 15798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:09:05.487356 15798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:05.489830 15798 catalog_manager.cc:1347] Generated new cluster ID: 93329c0e19c7421d862a9cf3ea66cf7b
I20260924 23:09:05.489946 15798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:09:05.493220 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.480797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.0.98.62:43791)
0924 23:09:05.481004 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:05.481015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:05.481053 (+ 38us) server_negotiation.cc:407] Connection header received
0924 23:09:05.481781 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:05.481791 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:05.481856 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:05.482027 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:05.482630 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.484398 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.485506 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.486192 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.486624 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.489428 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:05.489432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:05.489432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:05.489442 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:05.491523 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:05.491967 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:05.491970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:05.491970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:05.492072 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:05.492298 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:05.492299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:05.492299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:05.492584 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:05.492711 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:05.492898 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 23:09:05.493031 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260924 23:09:05.509227 15798 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:09:05.509789 15798 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:09:05.514439 15798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Generated new TSK 0
I20260924 23:09:05.514624 15798 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:09:05.550343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43791
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:05.645102 15803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:05.645364 15803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:05.645408 15803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:05.646932 15803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:05.647027 15803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:05.647063 15803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:09:05.648635 15803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:05.648831 15803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:05.649021 15803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:05.654397 15808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:05.654371 15811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:05.654430 15803 server_base.cc:1034] running on GCE node
W20260924 23:09:05.654384 15809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:05.654754 15803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:05.655025 15803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:05.656189 15803 hybrid_clock.cc:648] HybridClock initialized: now 1790291345656163 us; error 58 us; skew 500 ppm
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:05.658824 15803 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:09:05.659271 15803 webserver.cc:460] Webserver started at http://127.0.98.1:37301/ using document root <none> and password file <none>
I20260924 23:09:05.659497 15803 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:05.659561 15803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:05.659756 15803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:09:05.660854 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "127bc56fcfc247f18eb1f4612d2c2055"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "4f29c64887b00ba1df1f9a0d5e256a24"
server_key_iv: "adae857f8fc2d4a513c46728cd51cfbb"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.661219 15803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "127bc56fcfc247f18eb1f4612d2c2055"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "4f29c64887b00ba1df1f9a0d5e256a24"
server_key_iv: "adae857f8fc2d4a513c46728cd51cfbb"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.662683 15803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:09:05.663751 15818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.663933 15803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:05.664029 15803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "127bc56fcfc247f18eb1f4612d2c2055"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "4f29c64887b00ba1df1f9a0d5e256a24"
server_key_iv: "adae857f8fc2d4a513c46728cd51cfbb"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.664085 15803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:05.679771 15803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:05.681523 15803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:05.681634 15803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:05.681824 15803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:05.682170 15803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:05.682205 15803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.682228 15803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:05.682241 15803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.688227 15803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33589
I20260924 23:09:05.688290 15931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33589 every 8 connection(s)
I20260924 23:09:05.688622 15803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:05.695829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15803
I20260924 23:09:05.695912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:09:05.696144 392 external_mini_cluster.cc:1428] Setting key 6503ec62ad9a218bf535b027740f400e
I20260924 23:09:05.697417 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.689329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51125 (local address 127.0.98.62:43791)
0924 23:09:05.689373 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:05.689375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:05.689605 (+ 230us) server_negotiation.cc:407] Connection header received
0924 23:09:05.690093 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:05.690094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:05.690104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:05.690156 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:05.691039 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.691531 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.692475 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.692669 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.692718 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.694755 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:05.694756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:05.694756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:05.694762 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:05.695995 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:05.696464 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:05.696465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:05.696465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:05.696534 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:05.696785 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:05.696786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:05.696786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:05.696904 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:05.696973 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:05.697241 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 23:09:05.697328 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:09:05.697613 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.689320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:51125)
0924 23:09:05.689477 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:05.689512 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:05.690008 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:05.690208 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:05.690211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:05.690769 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:05.690960 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:05.690965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.691612 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.691613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:05.692399 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:05.692401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.692751 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.692752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:05.692846 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.693261 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:05.693269 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:05.694672 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:05.696065 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:05.696072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:05.696074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:05.696381 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:05.696604 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:05.696605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:05.696606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:05.696708 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:05.696974 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:05.696982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:05.697111 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 23:09:05.697213 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 23:09:05.697364 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":99,"threads_started":1}
I20260924 23:09:05.698060 15932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43791
I20260924 23:09:05.698153 15932 heartbeater.cc:464] Registering TS with master...
I20260924 23:09:05.698323 15932 heartbeater.cc:511] Master 127.0.98.62:43791 requested a full tablet report, sending...
I20260924 23:09:05.698848 15741 ts_manager.cc:194] Registered new tserver with Master: 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589)
I20260924 23:09:05.699471 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51125
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:09:05.721340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43791
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:05.803718 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:05.803938 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:05.803962 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:05.805050 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:05.805088 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:05.805114 15937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:09:05.806584 15937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:05.806770 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:05.806941 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:05.811807 15942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:05.812022 15937 server_base.cc:1034] running on GCE node
W20260924 23:09:05.811990 15943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:05.812176 15945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:05.812402 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:05.812611 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:05.813798 15937 hybrid_clock.cc:648] HybridClock initialized: now 1790291345813790 us; error 40 us; skew 500 ppm
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:05.816239 15937 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:09:05.816670 15937 webserver.cc:460] Webserver started at http://127.0.98.2:34405/ using document root <none> and password file <none>
I20260924 23:09:05.816912 15937 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:05.816982 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:05.817140 15937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:09:05.818286 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "79ecfb313e6245e285688d9f3216fb67"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "29a7f5bc6e9b3cfb98ae724bb551e5fc"
server_key_iv: "f4f4f6c7e873976e355f8be460ff8a1d"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.818646 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79ecfb313e6245e285688d9f3216fb67"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "29a7f5bc6e9b3cfb98ae724bb551e5fc"
server_key_iv: "f4f4f6c7e873976e355f8be460ff8a1d"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.819950 15937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:09:05.820699 15952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.820894 15937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:09:05.821009 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "79ecfb313e6245e285688d9f3216fb67"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "29a7f5bc6e9b3cfb98ae724bb551e5fc"
server_key_iv: "f4f4f6c7e873976e355f8be460ff8a1d"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.821082 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:05.833071 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:05.835397 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:05.835558 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:05.835789 15937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:05.836115 15937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:05.836170 15937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.836221 15937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:05.836252 15937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.842005 15937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34053
I20260924 23:09:05.842103 16065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34053 every 8 connection(s)
I20260924 23:09:05.842384 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:09:05.846150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 15937
I20260924 23:09:05.846232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:09:05.846489 392 external_mini_cluster.cc:1428] Setting key 038ddf9644b116d1b28458619f7bcfd6
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:05.852362 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.843118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43975 (local address 127.0.98.62:43791)
0924 23:09:05.843185 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:05.843187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:05.843405 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:09:05.843925 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:05.843926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:05.843938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:05.844001 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:05.844901 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.845483 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.846786 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.847115 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.847190 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.849372 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:05.849373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:05.849373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:05.849381 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:05.850982 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:05.851406 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:05.851407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:05.851408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:05.851466 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:05.851721 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:05.851722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:05.851722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:05.851838 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:05.851915 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:05.852187 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 23:09:05.852270 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:09:05.852576 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.843119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:43975)
0924 23:09:05.843287 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:05.843324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:05.843821 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:05.844072 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:05.844075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:05.844612 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:05.844804 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:05.844809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.845573 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.845574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:05.846703 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:05.846706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.847216 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.847217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:05.847327 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.847837 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:05.847845 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:05.849271 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:05.851084 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:05.851091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:05.851093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:05.851319 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:05.851539 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:05.851540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:05.851540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:05.851651 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:05.851906 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:05.851913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:05.852052 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 23:09:05.852169 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 23:09:05.852323 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260924 23:09:05.853183 16066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43791
I20260924 23:09:05.853279 16066 heartbeater.cc:464] Registering TS with master...
I20260924 23:09:05.853478 16066 heartbeater.cc:511] Master 127.0.98.62:43791 requested a full tablet report, sending...
I20260924 23:09:05.854001 15741 ts_manager.cc:194] Registered new tserver with Master: 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 23:09:05.854486 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:09:05.873131 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43791
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:05.953459 16071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:05.953675 16071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:05.953698 16071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:05.954910 16071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:05.954955 16071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:05.955015 16071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:09:05.956470 16071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:05.956717 16071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:05.956959 16071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:05.961804 16079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:05.961863 16076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:05.961808 16077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:05.962015 16071 server_base.cc:1034] running on GCE node
I20260924 23:09:05.962229 16071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:05.962469 16071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:05.963629 16071 hybrid_clock.cc:648] HybridClock initialized: now 1790291345963622 us; error 34 us; skew 500 ppm
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:05.966087 16071 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:09:05.966463 16071 webserver.cc:460] Webserver started at http://127.0.98.3:34945/ using document root <none> and password file <none>
I20260924 23:09:05.966698 16071 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:05.966742 16071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:05.966889 16071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:09:05.967969 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "ef66b857bcea793dfc129fe0cc74d717"
server_key_iv: "add84efdf822dbca9b1b49b4f7fc6640"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.968312 16071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "ef66b857bcea793dfc129fe0cc74d717"
server_key_iv: "add84efdf822dbca9b1b49b4f7fc6640"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.969570 16071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:09:05.970348 16086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.970528 16071 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.970615 16071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "ef66b857bcea793dfc129fe0cc74d717"
server_key_iv: "add84efdf822dbca9b1b49b4f7fc6640"
server_key_version: "encryptionkey@0"
I20260924 23:09:05.970687 16071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:05.983582 16071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:05.985968 16071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:05.986131 16071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:05.986404 16071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:05.986701 16071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:05.986732 16071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.986754 16071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:05.986779 16071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:05.992422 16071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39029
I20260924 23:09:05.992499 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39029 every 8 connection(s)
I20260924 23:09:05.992825 16071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:09:05.997963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16071
I20260924 23:09:05.998039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:09:05.998219 392 external_mini_cluster.cc:1428] Setting key c54c927d96c05317d638b5cae65efd3d
Sep 24 23:09:05 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:06.001950 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.993555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45377 (local address 127.0.98.62:43791)
0924 23:09:05.993605 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:05.993608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:05.993830 (+ 222us) server_negotiation.cc:407] Connection header received
0924 23:09:05.994324 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:05.994325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:05.994334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:05.994397 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:05.995261 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.995738 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.996673 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.996857 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.996905 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.999357 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:05.999359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:05.999359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:05.999367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:06.000527 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:06.000944 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:06.000946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:06.000946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:06.001002 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:06.001270 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:06.001271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:06.001271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:06.001388 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:06.001460 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.001736 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.001829 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:09:06.002140 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:05.993537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:45377)
0924 23:09:05.993709 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:05.993750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:05.994240 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:05.994449 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:05.994453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:05.994995 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:05.995187 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:05.995191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.995816 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.995816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:05.996602 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:05.996603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:05.996932 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:05.996933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:05.997034 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:05.997564 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:05.997572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:05.999257 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:06.000614 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:06.000621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:06.000624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:06.000854 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:06.001079 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:06.001080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:06.001080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:06.001192 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:06.001465 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:06.001472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:06.001608 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 23:09:06.001715 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 23:09:06.001871 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":96,"threads_started":1}
I20260924 23:09:06.002585 16200 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43791
I20260924 23:09:06.002691 16200 heartbeater.cc:464] Registering TS with master...
I20260924 23:09:06.002873 16200 heartbeater.cc:511] Master 127.0.98.62:43791 requested a full tablet report, sending...
I20260924 23:09:06.003229 15741 ts_manager.cc:194] Registered new tserver with Master: c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029)
I20260924 23:09:06.003604 15741 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45377
I20260924 23:09:06.010142 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:09:06.016878 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.011403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47656 (local address 127.0.98.62:43791)
0924 23:09:06.011442 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.011444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.011555 (+ 111us) server_negotiation.cc:407] Connection header received
0924 23:09:06.011604 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.011605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.011613 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.011662 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:06.012150 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.012598 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.013273 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.013444 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.013488 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.014666 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:06.014667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:06.014667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:06.014673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:06.015939 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:06.016242 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:06.016243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:06.016243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:06.016291 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:06.016419 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:06.016420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:06.016420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:06.016552 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:06.016626 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.016715 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.016782 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:09:06.018620 15741 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:09:06.020130 15741 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:09:06.026965 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.022464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33589 (local address 127.0.0.1:52952)
0924 23:09:06.022884 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:06.022893 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:06.022937 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:06.023468 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:06.023469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:06.023471 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:06.023513 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.023515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.024538 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.024539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.025670 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.025673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.026687 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.026688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.026764 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.026769 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:09:06.026822 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:09:06.026865 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":104,"thread_start_us":90,"threads_started":1}
I20260924 23:09:06.027048 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.022622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.98.3:39029)
0924 23:09:06.022816 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.022821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.022831 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:09:06.022874 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.022878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.022894 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.022978 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:06.023176 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.024343 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.025997 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.026488 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.026591 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.026650 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.026916 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.026958 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260924 23:09:06.027050 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.022656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.0.98.2:34053)
0924 23:09:06.023042 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.023046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.023056 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:09:06.023095 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.023097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.023111 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.023183 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:06.023325 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.024434 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.025984 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.026691 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.026758 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.026809 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.026913 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.026951 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":80,"threads_started":1}
I20260924 23:09:06.027345 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.022701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52952 (local address 127.0.98.1:33589)
0924 23:09:06.023323 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.023328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.023338 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:09:06.023377 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.023379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.023395 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.023470 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:06.023596 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.024441 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.025762 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.026520 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.026591 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.026650 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.027185 (+ 535us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.027228 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":92,"threads_started":1}
I20260924 23:09:06.027043 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.022447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34053 (local address 127.0.0.1:34618)
0924 23:09:06.022857 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:06.022866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:06.022902 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:06.023191 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:06.023192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:06.023194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:06.023238 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.023240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.024539 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.024539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.025864 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.025867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.026775 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.026776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.026831 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.026834 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:09:06.026896 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 23:09:06.026936 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":56,"thread_start_us":89,"threads_started":1}
I20260924 23:09:06.027408 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.022447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39029 (local address 127.0.0.1:50042)
0924 23:09:06.022616 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:06.022658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:06.022724 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:06.023001 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:06.023005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:06.023006 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:06.023079 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.023080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.024464 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.024465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.025865 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.025866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.026594 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.026595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.026684 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.026698 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:09:06.026778 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:09:06.026833 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":48,"thread_start_us":59,"threads_started":1}
I20260924 23:09:06.028189 16134 tablet_service.cc:1467] Processing CreateTablet for tablet da76fff03672468bb66509c80fad8b26 (DEFAULT_TABLE table=test-table [id=56ae2314685046ceaa787fc7bdecc2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:09:06.028192 15866 tablet_service.cc:1467] Processing CreateTablet for tablet da76fff03672468bb66509c80fad8b26 (DEFAULT_TABLE table=test-table [id=56ae2314685046ceaa787fc7bdecc2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:09:06.028518 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da76fff03672468bb66509c80fad8b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:06.028529 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da76fff03672468bb66509c80fad8b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:06.030345 16000 tablet_service.cc:1467] Processing CreateTablet for tablet da76fff03672468bb66509c80fad8b26 (DEFAULT_TABLE table=test-table [id=56ae2314685046ceaa787fc7bdecc2fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:09:06.030673 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da76fff03672468bb66509c80fad8b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:06.031260 16217 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Bootstrap starting.
I20260924 23:09:06.031626 16218 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap starting.
I20260924 23:09:06.031971 16217 tablet_bootstrap.cc:654] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:06.032284 16218 tablet_bootstrap.cc:654] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:06.032519 16217 log.cc:824] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:06.032836 16218 log.cc:824] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:06.033511 16217 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: No bootstrap required, opened a new log
I20260924 23:09:06.033589 16218 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: No bootstrap required, opened a new log
I20260924 23:09:06.033613 16217 ts_tablet_manager.cc:1397] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:06.033659 16218 ts_tablet_manager.cc:1397] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:06.034163 16221 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Bootstrap starting.
I20260924 23:09:06.034881 16221 tablet_bootstrap.cc:654] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:06.035220 16218 raft_consensus.cc:348] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.035372 16218 raft_consensus.cc:374] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:06.035394 16218 raft_consensus.cc:729] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127bc56fcfc247f18eb1f4612d2c2055, State: Initialized, Role: FOLLOWER
I20260924 23:09:06.035518 16218 consensus_queue.cc:260] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.035719 16221 log.cc:824] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:06.035820 16218 ts_tablet_manager.cc:1428] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:06.035869 15932 heartbeater.cc:503] Master 127.0.98.62:43791 was elected leader, sending a full tablet report...
I20260924 23:09:06.035796 16217 raft_consensus.cc:348] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.035949 16217 raft_consensus.cc:374] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:06.035983 16217 raft_consensus.cc:729] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cba8be0dc24b36a49c7d4b689a6888, State: Initialized, Role: FOLLOWER
I20260924 23:09:06.036089 16217 consensus_queue.cc:260] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.036371 16217 ts_tablet_manager.cc:1428] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 23:09:06.036484 16200 heartbeater.cc:503] Master 127.0.98.62:43791 was elected leader, sending a full tablet report...
I20260924 23:09:06.037395 16221 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: No bootstrap required, opened a new log
I20260924 23:09:06.037483 16221 ts_tablet_manager.cc:1397] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:09:06.038903 16221 raft_consensus.cc:348] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.039026 16221 raft_consensus.cc:374] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:06.039047 16221 raft_consensus.cc:729] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ecfb313e6245e285688d9f3216fb67, State: Initialized, Role: FOLLOWER
I20260924 23:09:06.039153 16221 consensus_queue.cc:260] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.039414 16221 ts_tablet_manager.cc:1428] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:06.039419 16066 heartbeater.cc:503] Master 127.0.98.62:43791 was elected leader, sending a full tablet report...
W20260924 23:09:06.093081 16067 tablet.cc:2369] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:09:06.167694 16222 raft_consensus.cc:482] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:09:06.167853 16222 raft_consensus.cc:504] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.168313 16222 leader_election.cc:288] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029), 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053)
I20260924 23:09:06.172261 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.168552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48645 (local address 127.0.98.2:34053)
0924 23:09:06.168602 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.168604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.168616 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:09:06.168705 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.168706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.168719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.168806 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:06.169061 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.169809 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.171337 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.171909 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.171983 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.171993 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.172152 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.172190 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:09:06.172261 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.168627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55817 (local address 127.0.98.3:39029)
0924 23:09:06.168683 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.168685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.168778 (+ 93us) server_negotiation.cc:407] Connection header received
0924 23:09:06.168834 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.168836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.168849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.168898 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:06.169069 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.169820 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.171336 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.171909 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.171983 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.171993 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.172152 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.172189 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:09:06.172303 15934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.168484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34053 (local address 127.0.98.1:48645)
0924 23:09:06.168539 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:06.168552 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:06.168608 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:06.168827 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:06.168828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:06.168829 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:06.168926 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.168929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.169921 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.169922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.171240 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.171244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.171995 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.171996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.172077 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.172079 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:09:06.172136 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:09:06.172189 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":59}
I20260924 23:09:06.172528 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.168491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39029 (local address 127.0.98.1:55817)
0924 23:09:06.168698 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:06.168710 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:06.168760 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:06.168920 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:06.168920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:06.168922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:06.168980 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.169000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.169921 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.169922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.171260 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:06.171261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.171995 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.171996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:06.172074 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.172079 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:09:06.172136 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:09:06.172189 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":168,"thread_start_us":61,"threads_started":1}
I20260924 23:09:06.172717 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da76fff03672468bb66509c80fad8b26" candidate_uuid: "127bc56fcfc247f18eb1f4612d2c2055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ecfb313e6245e285688d9f3216fb67" is_pre_election: true
I20260924 23:09:06.172916 16020 raft_consensus.cc:2454] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 127bc56fcfc247f18eb1f4612d2c2055 in term 0.
I20260924 23:09:06.172976 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da76fff03672468bb66509c80fad8b26" candidate_uuid: "127bc56fcfc247f18eb1f4612d2c2055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" is_pre_election: true
I20260924 23:09:06.173135 16154 raft_consensus.cc:2454] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 127bc56fcfc247f18eb1f4612d2c2055 in term 0.
I20260924 23:09:06.173157 15821 leader_election.cc:302] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055, 79ecfb313e6245e285688d9f3216fb67; no voters:
I20260924 23:09:06.173297 16222 raft_consensus.cc:2781] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:09:06.173367 16222 raft_consensus.cc:482] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:09:06.173393 16222 raft_consensus.cc:3037] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:06.174157 16222 raft_consensus.cc:504] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.174312 16222 leader_election.cc:288] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 1 election: Requested vote from peers c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029), 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053)
I20260924 23:09:06.174479 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da76fff03672468bb66509c80fad8b26" candidate_uuid: "127bc56fcfc247f18eb1f4612d2c2055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
I20260924 23:09:06.174567 16154 raft_consensus.cc:3037] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:06.175261 16154 raft_consensus.cc:2454] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 127bc56fcfc247f18eb1f4612d2c2055 in term 1.
I20260924 23:09:06.175406 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da76fff03672468bb66509c80fad8b26" candidate_uuid: "127bc56fcfc247f18eb1f4612d2c2055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ecfb313e6245e285688d9f3216fb67"
I20260924 23:09:06.175434 15820 leader_election.cc:302] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055, c3cba8be0dc24b36a49c7d4b689a6888; no voters:
I20260924 23:09:06.175491 16020 raft_consensus.cc:3037] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:06.175581 16222 raft_consensus.cc:2781] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:09:06.175757 16222 raft_consensus.cc:686] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 LEADER]: Becoming Leader. State: Replica: 127bc56fcfc247f18eb1f4612d2c2055, State: Running, Role: LEADER
I20260924 23:09:06.175881 16222 consensus_queue.cc:237] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:06.176386 16020 raft_consensus.cc:2454] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 127bc56fcfc247f18eb1f4612d2c2055 in term 1.
I20260924 23:09:06.176720 15740 catalog_manager.cc:5495] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "127bc56fcfc247f18eb1f4612d2c2055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } health_report { overall_health: HEALTHY } } }
W20260924 23:09:06.189345 15933 tablet.cc:2369] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:09:06.209859 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.207594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.0.98.1:33589)
0924 23:09:06.207658 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:06.207660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:06.207668 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:09:06.207712 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:06.207713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:06.207732 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:06.207809 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:09:06.207911 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.208363 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.209152 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:06.209333 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:06.209382 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:06.209515 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:09:06.209611 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:09:06.209671 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:06.209699 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 23:09:06.209715 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:09:06.211818 16154 raft_consensus.cc:1261] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Refusing update from remote peer 127bc56fcfc247f18eb1f4612d2c2055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:09:06.211818 16020 raft_consensus.cc:1261] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Refusing update from remote peer 127bc56fcfc247f18eb1f4612d2c2055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:09:06.212198 16227 consensus_queue.cc:1035] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:09:06.212337 16222 consensus_queue.cc:1035] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:09:06.215494 16232 mvcc.cc:205] Tried to move back new op lower bound from 7333033354081034240 to 7333033353937211392. Current Snapshot: MvccSnapshot[applied={T|T < 7333033354081034240}]
I20260924 23:09:06.215595 16231 mvcc.cc:205] Tried to move back new op lower bound from 7333033354081034240 to 7333033353937211392. Current Snapshot: MvccSnapshot[applied={T|T < 7333033354081034240 or (T in {7333033354081034240})}]
I20260924 23:09:06.216821 15740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:09:06.216930 15740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:09:06.217729 15740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Sending DeleteTablet for 3 replicas of tablet da76fff03672468bb66509c80fad8b26
I20260924 23:09:06.218073 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet da76fff03672468bb66509c80fad8b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34618
I20260924 23:09:06.218091 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet da76fff03672468bb66509c80fad8b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50042
I20260924 23:09:06.218400 15866 tablet_service.cc:1514] Processing DeleteTablet for tablet da76fff03672468bb66509c80fad8b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52952
I20260924 23:09:06.218473 16241 tablet_replica.cc:330] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: stopping tablet replica
I20260924 23:09:06.218480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 15803
I20260924 23:09:06.218566 16241 raft_consensus.cc:2229] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:09:06.218652 16241 raft_consensus.cc:2258] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:09:06.219120 16240 tablet_replica.cc:330] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: stopping tablet replica
I20260924 23:09:06.219214 16240 raft_consensus.cc:2229] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:09:06.219307 16240 raft_consensus.cc:2258] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:09:06.219518 16241 ts_tablet_manager.cc:1905] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:09:06.219692 16240 ts_tablet_manager.cc:1905] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:09:06.221427 16241 ts_tablet_manager.cc:1918] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:09:06.221427 16240 ts_tablet_manager.cc:1918] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:09:06.221504 16241 log.cc:1196] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/da76fff03672468bb66509c80fad8b26
I20260924 23:09:06.221504 16240 log.cc:1196] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/da76fff03672468bb66509c80fad8b26
I20260924 23:09:06.221782 16241 ts_tablet_manager.cc:1939] T da76fff03672468bb66509c80fad8b26 P c3cba8be0dc24b36a49c7d4b689a6888: Deleting consensus metadata
I20260924 23:09:06.221845 16240 ts_tablet_manager.cc:1939] T da76fff03672468bb66509c80fad8b26 P 79ecfb313e6245e285688d9f3216fb67: Deleting consensus metadata
I20260924 23:09:06.222222 15727 catalog_manager.cc:4841] TS 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053): tablet da76fff03672468bb66509c80fad8b26 (table test-table [id=56ae2314685046ceaa787fc7bdecc2fd]) successfully deleted
I20260924 23:09:06.222283 15726 catalog_manager.cc:4841] TS c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029): tablet da76fff03672468bb66509c80fad8b26 (table test-table [id=56ae2314685046ceaa787fc7bdecc2fd]) successfully deleted
W20260924 23:09:06.224157 15728 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33589 (error 108)
I20260924 23:09:06.224347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 15937
I20260924 23:09:06.225313 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:06.225185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33589 (local address 127.0.0.1:52968)
0924 23:09:06.225242 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:06.225278 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:09:06.225463 15728 catalog_manager.cc:4584] TS 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da76fff03672468bb66509c80fad8b26: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33589: connect: Connection refused (error 111)
I20260924 23:09:06.229228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 16071
I20260924 23:09:06.233625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 15709
I20260924 23:09:26.238858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43791
--webserver_interface=127.0.98.62
--webserver_port=41019
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:26.318536 16244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:26.318781 16244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:26.318818 16244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:26.319974 16244 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 23:09:26.320043 16244 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 23:09:26.320065 16244 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 23:09:26.320086 16244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:09:26.320115 16244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:26.320142 16244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:26.320171 16244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:09:26.320199 16244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:09:26.321657 16244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:26.321972 16244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:26.322202 16244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:26.330775 16250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.330780 16251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:26.330817 16244 server_base.cc:1034] running on GCE node
W20260924 23:09:26.330780 16253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:26.331236 16244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:26.331503 16244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:26.332639 16244 hybrid_clock.cc:648] HybridClock initialized: now 1790291366332625 us; error 27 us; skew 500 ppm
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:26.339263 16244 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:09:26.339747 16244 webserver.cc:460] Webserver started at http://127.0.98.62:41019/ using document root <none> and password file <none>
I20260924 23:09:26.340008 16244 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:26.340095 16244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:26.341560 16244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:09:26.342340 16260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.342555 16244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 23:09:26.342669 16244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "08589a56604d46bd977af36140d0f0e9"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "3a50a11a14712670a082dc8aa3738218"
server_key_iv: "736d9b0889edac30bc19156213df902c"
server_key_version: "encryptionkey@0"
I20260924 23:09:26.343107 16244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:26.354820 16244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:26.357028 16244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:26.357205 16244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:26.361521 16244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43791
I20260924 23:09:26.361583 16312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43791 every 8 connection(s)
I20260924 23:09:26.362067 16244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:09:26.363531 16313 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:09:26.363739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16244
I20260924 23:09:26.364003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33589
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43791
--webserver_port=37301
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:09:26.364374 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap starting.
I20260924 23:09:26.372617 16313 log.cc:824] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:26.374627 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:09:26.374918 16313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap complete.
I20260924 23:09:26.377158 16313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:26.377383 16313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08589a56604d46bd977af36140d0f0e9, State: Initialized, Role: FOLLOWER
I20260924 23:09:26.377554 16313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:26.377614 16313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:09:26.377681 16313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:09:26.377734 16313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 23:09:26.378859 16313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:26.378945 16313 leader_election.cc:302] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08589a56604d46bd977af36140d0f0e9; no voters:
I20260924 23:09:26.379158 16313 leader_election.cc:288] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 23:09:26.379192 16317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 23:09:26.379303 16317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 LEADER]: Becoming Leader. State: Replica: 08589a56604d46bd977af36140d0f0e9, State: Running, Role: LEADER
I20260924 23:09:26.379427 16313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:09:26.379446 16317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:26.379750 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08589a56604d46bd977af36140d0f0e9. Latest consensus state: current_term: 2 leader_uuid: "08589a56604d46bd977af36140d0f0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } } }
I20260924 23:09:26.379755 16318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "08589a56604d46bd977af36140d0f0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } } }
I20260924 23:09:26.379942 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:09:26.380020 16318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:09:26.380456 16324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:09:26.381253 16324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:09:26.381431 16324 catalog_manager.cc:1259] Loaded cluster ID: 93329c0e19c7421d862a9cf3ea66cf7b
I20260924 23:09:26.381582 16324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:09:26.382496 16324 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:09:26.382689 16324 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Loaded TSK: 0
I20260924 23:09:26.401393 16324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Generated new TSK 1
I20260924 23:09:26.401531 16324 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:09:26.456331 16315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:26.456578 16315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:26.456617 16315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:26.457769 16315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:26.457835 16315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:26.457880 16315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:09:26.459473 16315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37301
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:26.459736 16315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:26.459978 16315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:26.465986 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.465997 16343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.465997 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:26.466224 16315 server_base.cc:1034] running on GCE node
I20260924 23:09:26.466427 16315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:26.466660 16315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:26.467835 16315 hybrid_clock.cc:648] HybridClock initialized: now 1790291366467822 us; error 30 us; skew 500 ppm
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:26.470561 16315 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:09:26.471110 16315 webserver.cc:460] Webserver started at http://127.0.98.1:37301/ using document root <none> and password file <none>
I20260924 23:09:26.471378 16315 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:26.471463 16315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:26.473279 16315 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:26.474177 16350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.474403 16315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:26.474503 16315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "127bc56fcfc247f18eb1f4612d2c2055"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "4f29c64887b00ba1df1f9a0d5e256a24"
server_key_iv: "adae857f8fc2d4a513c46728cd51cfbb"
server_key_version: "encryptionkey@0"
I20260924 23:09:26.474946 16315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:26.493485 16315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:26.496364 16315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:26.496527 16315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:26.496795 16315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:26.497303 16357 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:09:26.498884 16315 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:09:26.498950 16315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:09:26.498986 16315 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:09:26.499945 16315 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:09:26.499992 16315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:09:26.500062 16357 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap starting.
I20260924 23:09:26.507252 16315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33589
I20260924 23:09:26.507398 16464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33589 every 8 connection(s)
I20260924 23:09:26.507779 16315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:09:26.509145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16315
I20260924 23:09:26.509424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43791
--webserver_port=34405
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:09:26.511134 16357 log.cc:824] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:26.514017 16357 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:09:26.514402 16357 tablet_bootstrap.cc:492] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap complete.
I20260924 23:09:26.515724 16357 ts_tablet_manager.cc:1397] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:26.517817 16357 raft_consensus.cc:348] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.518328 16357 raft_consensus.cc:729] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127bc56fcfc247f18eb1f4612d2c2055, State: Initialized, Role: FOLLOWER
I20260924 23:09:26.518558 16357 consensus_queue.cc:260] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.519253 16357 ts_tablet_manager.cc:1428] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:09:26.522414 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.508715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50977 (local address 127.0.98.62:43791)
0924 23:09:26.508924 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.508934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.509212 (+ 278us) server_negotiation.cc:407] Connection header received
0924 23:09:26.510242 (+ 1030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.510253 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.510325 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.510451 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:26.511870 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.513302 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.514897 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.515532 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.515615 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.518449 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:26.518453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:26.518453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:26.518465 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:26.520521 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.520975 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.520978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.520978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.521082 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.521345 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.521346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.521346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.521607 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:26.521674 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.522061 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.522206 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260924 23:09:26.522852 16467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.508614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:50977)
0924 23:09:26.509041 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:26.509088 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:26.509976 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:26.511101 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:26.511105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:26.511454 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:26.511689 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.511695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.513461 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.513462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.514786 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.514789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.515749 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.515750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.515893 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.516550 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:26.516560 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:26.518330 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:26.520610 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:26.520616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:26.520618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:26.520889 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:26.521150 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:26.521151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:26.521151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:26.521275 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:26.521708 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:26.521715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:26.521854 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 23:09:26.522506 (+ 652us) client_negotiation.cc:240] Negotiation successful
0924 23:09:26.522675 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":96,"threads_started":1}
I20260924 23:09:26.523510 16465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43791
I20260924 23:09:26.523602 16465 heartbeater.cc:464] Registering TS with master...
I20260924 23:09:26.523782 16465 heartbeater.cc:511] Master 127.0.98.62:43791 requested a full tablet report, sending...
I20260924 23:09:26.524531 16277 ts_manager.cc:194] Registered new tserver with Master: 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589)
I20260924 23:09:26.526547 16277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50977
I20260924 23:09:26.527628 16465 heartbeater.cc:503] Master 127.0.98.62:43791 was elected leader, sending a full tablet report...
I20260924 23:09:26.538146 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.525106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33589 (local address 127.0.0.1:54180)
0924 23:09:26.534007 (+ 8901us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:26.534049 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:26.534139 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:26.534387 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:26.534390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:26.534393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:26.534457 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.534459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.535706 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.535707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.536993 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.536996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.537809 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.537810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.537924 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.537939 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:09:26.538013 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:09:26.538066 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8866,"thread_start_us":103,"threads_started":1}
I20260924 23:09:26.538146 16473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.525358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.0.98.1:33589)
0924 23:09:26.534001 (+ 8643us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.534007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.534131 (+ 124us) server_negotiation.cc:407] Connection header received
0924 23:09:26.534217 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.534219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.534235 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.534317 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:26.534526 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.535606 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.537095 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.537716 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.537818 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.537875 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.538035 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.538076 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8606,"thread_start_us":84,"threads_started":1}
I20260924 23:09:26.538913 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet da76fff03672468bb66509c80fad8b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54180
I20260924 23:09:26.539572 16474 tablet_replica.cc:330] stopping tablet replica
I20260924 23:09:26.539681 16474 raft_consensus.cc:2229] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:09:26.539747 16474 raft_consensus.cc:2258] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:09:26.540081 16474 ts_tablet_manager.cc:1905] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:09:26.542785 16474 ts_tablet_manager.cc:1918] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:09:26.542861 16474 log.cc:1196] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal/wals/da76fff03672468bb66509c80fad8b26
I20260924 23:09:26.543084 16474 ts_tablet_manager.cc:1939] T da76fff03672468bb66509c80fad8b26 P 127bc56fcfc247f18eb1f4612d2c2055: Deleting consensus metadata
I20260924 23:09:26.543604 16264 catalog_manager.cc:4841] TS 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589): tablet da76fff03672468bb66509c80fad8b26 (table test-table [id=56ae2314685046ceaa787fc7bdecc2fd]) successfully deleted
W20260924 23:09:26.603688 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:26.603897 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:26.603920 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:26.605094 16469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:26.605136 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:26.605162 16469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:09:26.606621 16469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34405
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:26.606868 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:26.607100 16469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:26.612212 16482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.612250 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.612260 16480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:26.612408 16469 server_base.cc:1034] running on GCE node
I20260924 23:09:26.612598 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:26.612784 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:26.613945 16469 hybrid_clock.cc:648] HybridClock initialized: now 1790291366613927 us; error 30 us; skew 500 ppm
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:26.616367 16469 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:09:26.616814 16469 webserver.cc:460] Webserver started at http://127.0.98.2:34405/ using document root <none> and password file <none>
I20260924 23:09:26.617061 16469 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:26.617146 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:26.618589 16469 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:09:26.619200 16489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.619374 16469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:09:26.619475 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "79ecfb313e6245e285688d9f3216fb67"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "29a7f5bc6e9b3cfb98ae724bb551e5fc"
server_key_iv: "f4f4f6c7e873976e355f8be460ff8a1d"
server_key_version: "encryptionkey@0"
I20260924 23:09:26.619844 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:26.639497 16469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:26.641656 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:26.641839 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:26.642110 16469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:26.642426 16469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:26.642480 16469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.642524 16469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:26.642549 16469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.648698 16469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34053
I20260924 23:09:26.648746 16602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34053 every 8 connection(s)
I20260924 23:09:26.649088 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:26.656349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16469
I20260924 23:09:26.656622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39029
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43791
--webserver_port=34945
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:09:26.658490 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.649807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35073 (local address 127.0.98.62:43791)
0924 23:09:26.649862 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.649864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.650135 (+ 271us) server_negotiation.cc:407] Connection header received
0924 23:09:26.650612 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.650613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.650636 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.650709 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:26.651656 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.652170 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.653130 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.653328 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.653378 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.655470 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:26.655471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:26.655472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:26.655478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:26.656980 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.657458 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.657459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.657459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.657517 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.657787 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.657787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.657787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.657937 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:26.658018 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.658295 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.658398 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:09:26.658680 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.649804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:35073)
0924 23:09:26.649963 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:26.650007 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:26.650503 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:26.650779 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:26.650783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:26.651363 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:26.651558 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.651562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.652270 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.652270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.653062 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.653064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.653398 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.653399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.653499 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.654073 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:26.654081 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:26.655391 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:26.657054 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:26.657061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:26.657063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:26.657385 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:26.657580 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:26.657580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:26.657581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:26.657725 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:26.658011 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:26.658018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:26.658162 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 23:09:26.658275 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:09:26.658436 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":104,"threads_started":1}
I20260924 23:09:26.659324 16603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43791
I20260924 23:09:26.659425 16603 heartbeater.cc:464] Registering TS with master...
I20260924 23:09:26.659615 16603 heartbeater.cc:511] Master 127.0.98.62:43791 requested a full tablet report, sending...
I20260924 23:09:26.659976 16277 ts_manager.cc:194] Registered new tserver with Master: 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053)
I20260924 23:09:26.660358 16277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35073
W20260924 23:09:26.738600 16606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:26.738795 16606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:26.738819 16606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:26.739938 16606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:26.739976 16606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:26.740003 16606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:09:26.741490 16606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34945
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:26.741659 16606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:26.741830 16606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:26.746871 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.746866 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:26.746876 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:26.746969 16606 server_base.cc:1034] running on GCE node
I20260924 23:09:26.747251 16606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:26.747445 16606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:26.748585 16606 hybrid_clock.cc:648] HybridClock initialized: now 1790291366748571 us; error 28 us; skew 500 ppm
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:26.751151 16606 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:09:26.751663 16606 webserver.cc:460] Webserver started at http://127.0.98.3:34945/ using document root <none> and password file <none>
I20260924 23:09:26.751916 16606 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:26.751995 16606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:26.753307 16606 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:09:26.753988 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.754161 16606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:09:26.754259 16606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "ef66b857bcea793dfc129fe0cc74d717"
server_key_iv: "add84efdf822dbca9b1b49b4f7fc6640"
server_key_version: "encryptionkey@0"
I20260924 23:09:26.754614 16606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:26.802158 16606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:26.804204 16606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:26.804380 16606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:26.804641 16606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:26.804963 16606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:26.805023 16606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.805066 16606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:26.805096 16606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.811399 16606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39029
I20260924 23:09:26.811443 16734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39029 every 8 connection(s)
I20260924 23:09:26.811896 16606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:09:26.812433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16606
Sep 24 23:09:26 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:09:26.820576 16739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.812856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35812 (local address 127.0.98.62:43791)
0924 23:09:26.812976 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.812979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.813130 (+ 151us) server_negotiation.cc:407] Connection header received
0924 23:09:26.813183 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.813184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.813195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.813257 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:26.814140 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.814668 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.815568 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.815820 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.815871 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.817347 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:26.817348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:26.817348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:26.817353 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:26.819110 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.819490 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.819491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.819492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.819551 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.819854 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.819855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.819856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.820088 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:26.820300 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.820346 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.820439 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260924 23:09:26.822700 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.812697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54959 (local address 127.0.98.62:43791)
0924 23:09:26.812749 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.812752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.813008 (+ 256us) server_negotiation.cc:407] Connection header received
0924 23:09:26.813859 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.813860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.813867 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.813957 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:26.815045 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.815565 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.816966 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.817142 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.817197 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.819362 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:26.819363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:26.819363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:26.819370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:26.821216 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.821643 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.821644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.821644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.821705 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:26.822029 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:26.822030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:26.822031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:26.822145 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:26.822238 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.822523 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.822603 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:09:26.822942 16737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.812725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:54959)
0924 23:09:26.812878 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:26.812920 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:26.813687 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:26.814054 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:26.814059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:26.814649 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:26.814949 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.814956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.815658 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.815659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.816863 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.816867 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.817221 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.817222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.817376 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.817829 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:26.817836 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:26.819249 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:26.821302 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:26.821309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:26.821311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:26.821548 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:26.821779 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:26.821779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:26.821780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:26.821952 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:26.822233 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:26.822239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:26.822390 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 23:09:26.822503 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 23:09:26.822701 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260924 23:09:26.823581 16735 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43791
I20260924 23:09:26.823745 16735 heartbeater.cc:464] Registering TS with master...
I20260924 23:09:26.823938 16735 heartbeater.cc:511] Master 127.0.98.62:43791 requested a full tablet report, sending...
I20260924 23:09:26.824321 16277 ts_manager.cc:194] Registered new tserver with Master: c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029)
I20260924 23:09:26.824690 16277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54959
I20260924 23:09:26.832203 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:09:26.835307 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.832595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35826 (local address 127.0.98.62:43791)
0924 23:09:26.832671 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.832673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.832738 (+ 65us) server_negotiation.cc:407] Connection header received
0924 23:09:26.832786 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.832787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.832821 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.832898 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:09:26.833059 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.833777 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.834610 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.834802 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.834866 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.834993 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:09:26.835098 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:09:26.835160 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.835181 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.835198 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 23:09:26.835609 16277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:09:26.837287 16277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:09:26.839890 16385 tablet_service.cc:1467] Processing CreateTablet for tablet 2c40009a2f7e401aa7c36a591834b907 (DEFAULT_TABLE table=test-table [id=46f45c7a70bd4d2ca187ce4e643c1b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:09:26.840099 16385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c40009a2f7e401aa7c36a591834b907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:26.841445 16357 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap starting.
I20260924 23:09:26.841774 16357 tablet_bootstrap.cc:654] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:26.842654 16357 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: No bootstrap required, opened a new log
I20260924 23:09:26.842706 16357 ts_tablet_manager.cc:1397] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 23:09:26.842828 16357 raft_consensus.cc:348] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.842921 16357 raft_consensus.cc:374] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:26.842979 16357 raft_consensus.cc:729] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127bc56fcfc247f18eb1f4612d2c2055, State: Initialized, Role: FOLLOWER
I20260924 23:09:26.843039 16357 consensus_queue.cc:260] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.843146 16357 ts_tablet_manager.cc:1428] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:26.844184 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.839487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39029 (local address 127.0.0.1:45798)
0924 23:09:26.839692 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:26.839703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:26.839756 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:26.840090 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:26.840091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:26.840093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:26.840150 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.840152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.841162 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.841170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.842236 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.842239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.843959 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.843959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.844020 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.844024 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:09:26.844084 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 23:09:26.844123 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":139,"threads_started":1}
I20260924 23:09:26.844216 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.839768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.0.98.3:39029)
0924 23:09:26.839936 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.839941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.839950 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:09:26.839989 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.839992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.840015 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.840092 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:26.840226 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.841073 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.842367 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.843877 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.843960 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.844027 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.844097 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.844137 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260924 23:09:26.844978 16669 tablet_service.cc:1467] Processing CreateTablet for tablet 2c40009a2f7e401aa7c36a591834b907 (DEFAULT_TABLE table=test-table [id=46f45c7a70bd4d2ca187ce4e643c1b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:09:26.845150 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.839455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34053 (local address 127.0.0.1:42650)
0924 23:09:26.839576 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:26.839588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:26.839633 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:26.841687 (+ 2054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:26.841688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:26.841690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:26.841731 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.841733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.842926 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.842927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.844002 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:26.844004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.844925 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.844927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:26.844989 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.844992 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:09:26.845057 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:09:26.845095 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 23:09:26.845206 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:26.839601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.0.98.2:34053)
0924 23:09:26.841520 (+ 1919us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:26.841525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:26.841535 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:09:26.841575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:26.841577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:26.841591 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:26.841691 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:26.841788 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.842852 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.844072 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:26.844834 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:26.844902 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:26.844956 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:26.845076 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 23:09:26.845118 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1879,"thread_start_us":81,"threads_started":1}
I20260924 23:09:26.845366 16669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c40009a2f7e401aa7c36a591834b907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:26.848366 16745 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Bootstrap starting.
I20260924 23:09:26.848803 16537 tablet_service.cc:1467] Processing CreateTablet for tablet 2c40009a2f7e401aa7c36a591834b907 (DEFAULT_TABLE table=test-table [id=46f45c7a70bd4d2ca187ce4e643c1b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:09:26.848999 16745 tablet_bootstrap.cc:654] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:26.849134 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c40009a2f7e401aa7c36a591834b907. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:09:26.849553 16745 log.cc:824] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:26.850315 16745 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: No bootstrap required, opened a new log
I20260924 23:09:26.850416 16745 ts_tablet_manager.cc:1397] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:26.851976 16745 raft_consensus.cc:348] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.852106 16747 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Bootstrap starting.
I20260924 23:09:26.852142 16745 raft_consensus.cc:374] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:26.852183 16745 raft_consensus.cc:729] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3cba8be0dc24b36a49c7d4b689a6888, State: Initialized, Role: FOLLOWER
I20260924 23:09:26.852305 16745 consensus_queue.cc:260] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.852614 16745 ts_tablet_manager.cc:1428] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:26.852643 16735 heartbeater.cc:503] Master 127.0.98.62:43791 was elected leader, sending a full tablet report...
I20260924 23:09:26.852795 16747 tablet_bootstrap.cc:654] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Neither blocks nor log segments found. Creating new log.
I20260924 23:09:26.853322 16747 log.cc:824] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:26.854120 16747 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: No bootstrap required, opened a new log
I20260924 23:09:26.854199 16747 ts_tablet_manager.cc:1397] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:26.855691 16747 raft_consensus.cc:348] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.855867 16747 raft_consensus.cc:374] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:09:26.855922 16747 raft_consensus.cc:729] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ecfb313e6245e285688d9f3216fb67, State: Initialized, Role: FOLLOWER
I20260924 23:09:26.856096 16747 consensus_queue.cc:260] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:26.856397 16747 ts_tablet_manager.cc:1428] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:09:26.856401 16603 heartbeater.cc:503] Master 127.0.98.62:43791 was elected leader, sending a full tablet report...
W20260924 23:09:26.899652 16604 tablet.cc:2369] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:09:27.008888 16466 tablet.cc:2369] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:09:27.062501 16736 tablet.cc:2369] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:09:27.146492 16750 raft_consensus.cc:482] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:09:27.146644 16750 raft_consensus.cc:504] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:27.147061 16750 leader_election.cc:288] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029), 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589)
I20260924 23:09:27.149977 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.147362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38997 (local address 127.0.98.1:33589)
0924 23:09:27.147489 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:27.147491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:27.147499 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:09:27.147531 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:27.147532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:27.147548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:27.147621 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:27.147776 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.148277 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.149213 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.149590 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.149668 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.149680 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:27.149824 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 23:09:27.149899 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260924 23:09:27.150005 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.147408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53163 (local address 127.0.98.3:39029)
0924 23:09:27.147459 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:27.147461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:27.147524 (+ 63us) server_negotiation.cc:407] Connection header received
0924 23:09:27.147604 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:27.147605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:27.147624 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:27.147691 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:09:27.147863 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.148326 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.149250 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.149621 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.149717 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.149727 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:27.149872 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 23:09:27.149927 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 23:09:27.150038 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.147328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39029 (local address 127.0.98.2:53163)
0924 23:09:27.147465 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:27.147476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:27.147539 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:27.147696 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:27.147697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:27.147699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:27.147776 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:27.147778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.148416 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.148417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:27.149149 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:27.149151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.149712 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.149713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:27.149793 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.149797 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:09:27.149866 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 23:09:27.149927 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260924 23:09:27.150127 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.147307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33589 (local address 127.0.98.2:38997)
0924 23:09:27.147385 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:27.147398 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:27.147446 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:27.147624 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:27.147625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:27.147627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:09:27.147703 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:27.147705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.148370 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.148371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:27.149115 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:27.149117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.149666 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.149667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:27.149749 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.149752 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:09:27.149819 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 23:09:27.149858 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 23:09:27.150329 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c40009a2f7e401aa7c36a591834b907" candidate_uuid: "79ecfb313e6245e285688d9f3216fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" is_pre_election: true
I20260924 23:09:27.150485 16689 raft_consensus.cc:2454] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ecfb313e6245e285688d9f3216fb67 in term 0.
I20260924 23:09:27.150490 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c40009a2f7e401aa7c36a591834b907" candidate_uuid: "79ecfb313e6245e285688d9f3216fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "127bc56fcfc247f18eb1f4612d2c2055" is_pre_election: true
I20260924 23:09:27.150656 16419 raft_consensus.cc:2454] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ecfb313e6245e285688d9f3216fb67 in term 0.
I20260924 23:09:27.150842 16491 leader_election.cc:302] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ecfb313e6245e285688d9f3216fb67, c3cba8be0dc24b36a49c7d4b689a6888; no voters:
I20260924 23:09:27.150993 16750 raft_consensus.cc:2781] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:09:27.151067 16750 raft_consensus.cc:482] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:09:27.151086 16750 raft_consensus.cc:3037] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:27.151830 16750 raft_consensus.cc:504] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:27.151973 16750 leader_election.cc:288] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [CANDIDATE]: Term 1 election: Requested vote from peers c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029), 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589)
I20260924 23:09:27.152117 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c40009a2f7e401aa7c36a591834b907" candidate_uuid: "79ecfb313e6245e285688d9f3216fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "127bc56fcfc247f18eb1f4612d2c2055"
I20260924 23:09:27.152202 16419 raft_consensus.cc:3037] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:27.152172 16689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c40009a2f7e401aa7c36a591834b907" candidate_uuid: "79ecfb313e6245e285688d9f3216fb67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
I20260924 23:09:27.152281 16689 raft_consensus.cc:3037] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:09:27.152885 16419 raft_consensus.cc:2454] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ecfb313e6245e285688d9f3216fb67 in term 1.
I20260924 23:09:27.153056 16493 leader_election.cc:302] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055, 79ecfb313e6245e285688d9f3216fb67; no voters:
I20260924 23:09:27.153160 16750 raft_consensus.cc:2781] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:09:27.153216 16689 raft_consensus.cc:2454] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ecfb313e6245e285688d9f3216fb67 in term 1.
I20260924 23:09:27.153302 16750 raft_consensus.cc:686] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 1 LEADER]: Becoming Leader. State: Replica: 79ecfb313e6245e285688d9f3216fb67, State: Running, Role: LEADER
I20260924 23:09:27.153417 16750 consensus_queue.cc:237] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:27.154382 16277 catalog_manager.cc:5495] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "79ecfb313e6245e285688d9f3216fb67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } health_report { overall_health: UNKNOWN } } }
I20260924 23:09:27.196436 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.194183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.0.98.2:34053)
0924 23:09:27.194251 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:27.194253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:27.194261 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:09:27.194298 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:27.194299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:27.194321 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:27.194393 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:09:27.194531 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.195052 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.195777 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.195970 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.196024 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.196166 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:09:27.196235 (+ 69us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 23:09:27.196363 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42654: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 23:09:27.196527 16742 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42654: token expired
I20260924 23:09:27.202998 16468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.196852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.0.98.62:43791)
0924 23:09:27.196919 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:27.196925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:27.196933 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:09:27.197006 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:27.197006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:27.197017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:27.197070 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:27.197584 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.198244 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.198897 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.199091 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.199149 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.200323 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:27.200324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:27.200325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:27.200331 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:27.201769 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:27.202151 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:27.202151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:27.202152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:27.202235 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:27.202404 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:27.202404 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:27.202404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:27.202562 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:27.202688 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:27.202867 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 23:09:27.202932 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:09:27.206156 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.203907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42656 (local address 127.0.98.2:34053)
0924 23:09:27.203961 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:27.203963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:27.203972 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:09:27.204000 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:27.204001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:27.204012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:27.204070 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:09:27.204169 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.204625 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.205358 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:27.205542 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:27.205624 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:27.205779 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:09:27.205913 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:09:27.205983 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:27.206056 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:09:27.206078 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:09:27.208024 16689 raft_consensus.cc:1261] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Refusing update from remote peer 79ecfb313e6245e285688d9f3216fb67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:09:27.208021 16419 raft_consensus.cc:1261] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Refusing update from remote peer 79ecfb313e6245e285688d9f3216fb67: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:09:27.208431 16753 consensus_queue.cc:1035] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:09:27.208588 16750 consensus_queue.cc:1035] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:09:27.210938 16758 mvcc.cc:205] Tried to move back new op lower bound from 7333033440081272832 to 7333033439861112832. Current Snapshot: MvccSnapshot[applied={T|T < 7333033440081272832}]
I20260924 23:09:27.211388 16757 mvcc.cc:205] Tried to move back new op lower bound from 7333033440081272832 to 7333033439861112832. Current Snapshot: MvccSnapshot[applied={T|T < 7333033440081272832}]
I20260924 23:09:27.212882 16277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:09:27.212968 16277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35842:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:09:27.213639 16277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Sending DeleteTablet for 3 replicas of tablet 2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:27.213976 16537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c40009a2f7e401aa7c36a591834b907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42650
I20260924 23:09:27.213992 16669 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c40009a2f7e401aa7c36a591834b907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45798
I20260924 23:09:27.214185 16385 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c40009a2f7e401aa7c36a591834b907 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:09:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54180
I20260924 23:09:27.214208 16765 tablet_replica.cc:330] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: stopping tablet replica
I20260924 23:09:27.214320 16765 raft_consensus.cc:2229] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:09:27.214373 16767 tablet_replica.cc:330] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: stopping tablet replica
I20260924 23:09:27.214447 16765 raft_consensus.cc:2258] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:09:27.214481 16767 raft_consensus.cc:2229] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:09:27.214516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 16315
I20260924 23:09:27.214677 16767 raft_consensus.cc:2258] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:09:27.215086 16765 ts_tablet_manager.cc:1905] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:09:27.216769 16765 ts_tablet_manager.cc:1918] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:09:27.216846 16765 log.cc:1196] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:27.217099 16765 ts_tablet_manager.cc:1939] T 2c40009a2f7e401aa7c36a591834b907 P 79ecfb313e6245e285688d9f3216fb67: Deleting consensus metadata
I20260924 23:09:27.217516 16263 catalog_manager.cc:4841] TS 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053): tablet 2c40009a2f7e401aa7c36a591834b907 (table test-table [id=46f45c7a70bd4d2ca187ce4e643c1b6c]) successfully deleted
I20260924 23:09:27.217995 16766 tablet_replica.cc:330] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: stopping tablet replica
I20260924 23:09:27.218112 16766 raft_consensus.cc:2229] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:09:27.218225 16766 raft_consensus.cc:2258] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:09:27.219107 16766 ts_tablet_manager.cc:1905] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:09:27.220590 16766 ts_tablet_manager.cc:1918] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 23:09:27.220655 16264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33589 (error 108)
I20260924 23:09:27.220745 16766 log.cc:1196] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:27.221067 16766 ts_tablet_manager.cc:1939] T 2c40009a2f7e401aa7c36a591834b907 P c3cba8be0dc24b36a49c7d4b689a6888: Deleting consensus metadata
I20260924 23:09:27.221102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 16469
I20260924 23:09:27.222059 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:27.221873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33589 (local address 127.0.0.1:54190)
0924 23:09:27.221980 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:27.222021 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260924 23:09:27.222170 16264 catalog_manager.cc:4584] TS 127bc56fcfc247f18eb1f4612d2c2055 (127.0.98.1:33589): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c40009a2f7e401aa7c36a591834b907: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33589: connect: Connection refused (error 111)
I20260924 23:09:27.222570 16262 catalog_manager.cc:4841] TS c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029): tablet 2c40009a2f7e401aa7c36a591834b907 (table test-table [id=46f45c7a70bd4d2ca187ce4e643c1b6c]) successfully deleted
I20260924 23:09:27.226737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 16606
I20260924 23:09:27.231158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 16244
I20260924 23:09:47.236687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43791
--webserver_interface=127.0.98.62
--webserver_port=41019
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:47.316774 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:47.316991 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:47.317013 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:47.318261 16769 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 23:09:47.318301 16769 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 23:09:47.318315 16769 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 23:09:47.318326 16769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:09:47.318383 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:47.318400 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:47.318427 16769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:09:47.318465 16769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:09:47.319962 16769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:47.320231 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:47.320470 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:47.325418 16775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:47.325704 16774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.325742 16769 server_base.cc:1034] running on GCE node
W20260924 23:09:47.326188 16777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.326541 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:47.326857 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:47.328056 16769 hybrid_clock.cc:648] HybridClock initialized: now 1790291387328029 us; error 44 us; skew 500 ppm
Sep 24 23:09:47 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:47.337451 16769 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:09:47.338075 16769 webserver.cc:460] Webserver started at http://127.0.98.62:41019/ using document root <none> and password file <none>
I20260924 23:09:47.338327 16769 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:47.338400 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:47.339838 16769 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:09:47.340590 16784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.340813 16769 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:09:47.340920 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "08589a56604d46bd977af36140d0f0e9"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "3a50a11a14712670a082dc8aa3738218"
server_key_iv: "736d9b0889edac30bc19156213df902c"
server_key_version: "encryptionkey@0"
I20260924 23:09:47.341300 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:47.364197 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:47.366173 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:47.366338 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:47.370285 16769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43791
I20260924 23:09:47.370319 16836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43791 every 8 connection(s)
I20260924 23:09:47.370711 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:09:47.371788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16769
I20260924 23:09:47.372043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33589
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43791
--webserver_port=37301
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:09:47.372308 16837 sys_catalog.cc:263] Verifying existing consensus state
I20260924 23:09:47.372957 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap starting.
I20260924 23:09:47.384002 16837 log.cc:824] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:47.387017 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:09:47.387379 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Bootstrap complete.
I20260924 23:09:47.390712 16837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:47.391044 16837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08589a56604d46bd977af36140d0f0e9, State: Initialized, Role: FOLLOWER
I20260924 23:09:47.391230 16837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:47.391297 16837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:09:47.391338 16837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:09:47.391394 16837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 2 FOLLOWER]: Advancing to term 3
I20260924 23:09:47.392876 16837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:47.392992 16837 leader_election.cc:302] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08589a56604d46bd977af36140d0f0e9; no voters:
I20260924 23:09:47.393172 16837 leader_election.cc:288] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 23:09:47.393262 16841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 23:09:47.393419 16837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:09:47.393476 16841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [term 3 LEADER]: Becoming Leader. State: Replica: 08589a56604d46bd977af36140d0f0e9, State: Running, Role: LEADER
I20260924 23:09:47.393640 16841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } }
I20260924 23:09:47.394037 16842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "08589a56604d46bd977af36140d0f0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } } }
I20260924 23:09:47.394129 16842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:09:47.394294 16843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08589a56604d46bd977af36140d0f0e9. Latest consensus state: current_term: 3 leader_uuid: "08589a56604d46bd977af36140d0f0e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08589a56604d46bd977af36140d0f0e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43791 } } }
I20260924 23:09:47.394373 16843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:09:47.394318 16850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:09:47.395205 16850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:09:47.395304 16850 catalog_manager.cc:1259] Loaded cluster ID: 93329c0e19c7421d862a9cf3ea66cf7b
I20260924 23:09:47.395336 16850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:09:47.396516 16850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:09:47.396704 16850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Loaded TSK: 0
I20260924 23:09:47.396742 16850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08589a56604d46bd977af36140d0f0e9: Loaded TSK: 1
F20260924 23:09:47.396955 16850 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd400b619ad google::LogMessage::Fail() at ??:0
@ 0x7fd400b6488a google::LogMessage::SendToLog() at ??:0
@ 0x7fd400b614de google::LogMessage::Flush() at ??:0
@ 0x7fd400b62a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd4032ce104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd4032cbd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd408abb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd408abb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd408abb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd408a81c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd408a8bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd402a2f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd402a24954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd4038886db start_thread at ??:0
@ 0x7fd3ff65f61f clone at ??:0
W20260924 23:09:47.474807 16839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:47.475034 16839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:47.475071 16839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:47.476203 16839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:47.476269 16839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:47.476316 16839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:09:47.477849 16839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37301
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:47.478165 16839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:47.478392 16839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:47.483842 16879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.483940 16839 server_base.cc:1034] running on GCE node
W20260924 23:09:47.483834 16881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:47.483922 16878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.484400 16839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:47.484714 16839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:47.485922 16839 hybrid_clock.cc:648] HybridClock initialized: now 1790291387485907 us; error 42 us; skew 500 ppm
Sep 24 23:09:47 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:47.489197 16839 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:09:47.489804 16839 webserver.cc:460] Webserver started at http://127.0.98.1:37301/ using document root <none> and password file <none>
I20260924 23:09:47.490121 16839 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:47.490193 16839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:47.492156 16839 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:09:47.493021 16888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.493242 16839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:47.493352 16839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "127bc56fcfc247f18eb1f4612d2c2055"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "4f29c64887b00ba1df1f9a0d5e256a24"
server_key_iv: "adae857f8fc2d4a513c46728cd51cfbb"
server_key_version: "encryptionkey@0"
I20260924 23:09:47.493834 16839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:47.510825 16839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:47.513465 16839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:47.513638 16839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:47.513981 16839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:47.514539 16895 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 23:09:47.516070 16839 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 23:09:47.516141 16839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 23:09:47.516178 16839 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 23:09:47.517146 16839 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 23:09:47.517194 16839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 23:09:47.517273 16895 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap starting.
I20260924 23:09:47.525087 16839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33589
I20260924 23:09:47.525169 17002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33589 every 8 connection(s)
I20260924 23:09:47.525640 16839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:09:47.527823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 16839
I20260924 23:09:47.528121 16895 log.cc:824] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Log is configured to *not* fsync() on all Append() calls
I20260924 23:09:47.528146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34053
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43791
--webserver_port=34405
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:09:47.530659 16895 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 23:09:47.530958 16895 tablet_bootstrap.cc:492] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Bootstrap complete.
I20260924 23:09:47.532159 16895 ts_tablet_manager.cc:1397] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260924 23:09:47.534045 16895 raft_consensus.cc:348] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:47.534439 16895 raft_consensus.cc:729] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 127bc56fcfc247f18eb1f4612d2c2055, State: Initialized, Role: FOLLOWER
I20260924 23:09:47.534643 16895 consensus_queue.cc:260] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:47.534946 16895 ts_tablet_manager.cc:1428] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055: Time spent starting tablet: real 0.003s user 0.003s sys 0.002s
W20260924 23:09:47.612741 17007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:47.612949 17007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:47.612977 17007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:47.614310 17007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:47.614367 17007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:47.614419 17007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:09:47.617442 17007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34405
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:47.617717 17007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:47.618021 17007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:47.619717 17005 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:09:47.526447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:56857)
0924 23:09:47.526683 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.526742 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:47.527516 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:47.619473 (+ 91957us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":116,"threads_started":1}
W20260924 23:09:47.619976 17003 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43791: Client connection negotiation failed: client connection to 127.0.98.62:43791: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 23:09:47.620420 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.620271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:33943)
0924 23:09:47.620341 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.620389 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 23:09:47.620780 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.620668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:37239)
0924 23:09:47.620719 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.620753 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 23:09:47.620908 17003 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 23:09:47.623970 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:47.623972 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.624181 17007 server_base.cc:1034] running on GCE node
W20260924 23:09:47.624043 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.624511 17007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:47.624775 17007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:47.625950 17007 hybrid_clock.cc:648] HybridClock initialized: now 1790291387625927 us; error 46 us; skew 500 ppm
Sep 24 23:09:47 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:47.628475 17007 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:09:47.628916 17007 webserver.cc:460] Webserver started at http://127.0.98.2:34405/ using document root <none> and password file <none>
I20260924 23:09:47.629168 17007 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:47.629246 17007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:47.630676 17007 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:47.631309 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.631508 17007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:47.631618 17007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "79ecfb313e6245e285688d9f3216fb67"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "29a7f5bc6e9b3cfb98ae724bb551e5fc"
server_key_iv: "f4f4f6c7e873976e355f8be460ff8a1d"
server_key_version: "encryptionkey@0"
I20260924 23:09:47.631964 17007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:47.655190 17007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:47.657261 17007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:47.657428 17007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:47.657670 17007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:47.658064 17007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:47.658119 17007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.658162 17007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:47.658190 17007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.664254 17007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34053
I20260924 23:09:47.664335 17136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34053 every 8 connection(s)
I20260924 23:09:47.664691 17007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:09:47.665858 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.665375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:59761)
0924 23:09:47.665543 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.665669 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
W20260924 23:09:47.666326 17137 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43791: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
I20260924 23:09:47.666601 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.666506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:41913)
0924 23:09:47.666555 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.666583 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:09:47.666934 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.666781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:43417)
0924 23:09:47.666878 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.666911 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 23:09:47.667088 17137 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:09:47.674441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17007
I20260924 23:09:47.674711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39029
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43791
--webserver_port=34945
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:09:47.758491 17140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:09:47.758684 17140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:09:47.758706 17140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:09:47.759764 17140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:09:47.759805 17140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:09:47.759830 17140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:09:47.761159 17140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39029
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34945
--tserver_master_addrs=127.0.98.62:43791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:09:47.761345 17140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:09:47.761511 17140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:09:47.766783 17145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:47.766703 17146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:09:47.766687 17148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:09:47.766829 17140 server_base.cc:1034] running on GCE node
I20260924 23:09:47.767159 17140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:09:47.767400 17140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:09:47.768563 17140 hybrid_clock.cc:648] HybridClock initialized: now 1790291387768558 us; error 39 us; skew 500 ppm
Sep 24 23:09:47 dist-test-slave-ph9j krb5kdc[15696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:09:47.771066 17140 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:09:47.771560 17140 webserver.cc:460] Webserver started at http://127.0.98.3:34945/ using document root <none> and password file <none>
I20260924 23:09:47.771786 17140 fs_manager.cc:359] Metadata directory not provided
I20260924 23:09:47.771832 17140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:09:47.773213 17140 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:47.773859 17155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.774097 17140 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:09:47.774194 17140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "c3cba8be0dc24b36a49c7d4b689a6888"
format_stamp: "Formatted at 2026-09-24 23:09:05 on dist-test-slave-ph9j"
server_key: "ef66b857bcea793dfc129fe0cc74d717"
server_key_iv: "add84efdf822dbca9b1b49b4f7fc6640"
server_key_version: "encryptionkey@0"
I20260924 23:09:47.774569 17140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:09:47.799782 17140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:09:47.802114 17140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:09:47.802299 17140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:09:47.802590 17140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:09:47.802948 17140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:09:47.802986 17140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.803011 17140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:09:47.803066 17140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:09:47.809230 17140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39029
I20260924 23:09:47.809283 17268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39029 every 8 connection(s)
I20260924 23:09:47.809646 17140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:09:47.810786 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.810377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:45469)
0924 23:09:47.810530 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.810639 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260924 23:09:47.811194 17269 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43791 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43791: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
I20260924 23:09:47.811496 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.811408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:42185)
0924 23:09:47.811444 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.811477 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 23:09:47.811805 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:47.811707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:44773)
0924 23:09:47.811757 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:47.811783 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 23:09:47.811965 17269 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 23:09:47.819908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17140
I20260924 23:09:48.621481 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.621255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:49421)
0924 23:09:48.621374 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:48.621414 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 23:09:48.667601 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.667367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:40591)
0924 23:09:48.667512 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:48.667548 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":58,"threads_started":1}
I20260924 23:09:48.812539 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.812332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:43709)
0924 23:09:48.812451 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:48.812485 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260924 23:09:48.896799 17276 raft_consensus.cc:482] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:09:48.896916 17276 raft_consensus.cc:504] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:48.897315 17276 leader_election.cc:288] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053), c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029)
Sep 24 23:09:48 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 23:09:48 dist-test-slave-ph9j krb5kdc[15696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 23:09:48.907688 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.897630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49703 (local address 127.0.98.2:34053)
0924 23:09:48.897788 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:48.897800 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:48.897840 (+ 40us) server_negotiation.cc:407] Connection header received
0924 23:09:48.898385 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:48.898388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:48.898425 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:48.898578 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:48.899305 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.900173 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.901178 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.901577 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.901700 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:48.904159 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:48.904163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:48.904163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:48.904176 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:48.906065 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906498 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:48.906501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:48.906501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:48.906592 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906846 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:48.906847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:48.906848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:48.907077 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:48.907164 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:48.907456 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 23:09:48.907571 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260924 23:09:48.907680 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.897511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34053 (local address 127.0.98.1:49703)
0924 23:09:48.897659 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:48.897671 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:48.897726 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:48.898611 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:48.898614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:48.899000 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:48.899185 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:48.899191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.900296 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.900298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:48.901077 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:48.901079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.901679 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.901680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:48.901796 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:48.902460 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:48.902471 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:48.904008 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:48.906162 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906169 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:48.906171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:48.906397 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:48.906670 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:48.906671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:48.906776 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:48.907158 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:48.907164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:48.907296 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 23:09:48.907407 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 23:09:48.907586 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":27,"thread_start_us":74,"threads_started":1}
I20260924 23:09:48.907780 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.897609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44505 (local address 127.0.98.3:39029)
0924 23:09:48.897829 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 23:09:48.897840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 23:09:48.897910 (+ 70us) server_negotiation.cc:407] Connection header received
0924 23:09:48.898352 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:09:48.898354 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:09:48.898402 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:09:48.898560 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:09:48.899308 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.900173 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.901172 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.901629 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.901742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:48.904446 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:09:48.904451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:09:48.904451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:09:48.904463 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:09:48.906357 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906707 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:48.906710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:48.906710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:48.906799 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:09:48.907024 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:09:48.907025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:09:48.907025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:09:48.907240 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:09:48.907327 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:09:48.907524 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 23:09:48.907662 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":54,"threads_started":1}
I20260924 23:09:48.907788 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:48.897472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39029 (local address 127.0.98.1:44505)
0924 23:09:48.897534 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:48.897545 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:09:48.897601 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:09:48.898610 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:09:48.898613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:09:48.899000 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:09:48.899185 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:48.899191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.900282 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.900283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:48.901076 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:09:48.901078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:09:48.901710 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:09:48.901711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:09:48.901788 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:09:48.902464 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:09:48.902476 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:09:48.904336 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:09:48.906443 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:48.906445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:48.906613 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:48.906876 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:09:48.906876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:09:48.906877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:09:48.906944 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:09:48.907323 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:09:48.907324 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:09:48.907367 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:09:48.907455 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:09:48.907668 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":153}
W20260924 23:09:48.908864 16890 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
W20260924 23:09:48.908959 16891 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:48.909035 16891 leader_election.cc:302] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055; no voters: 79ecfb313e6245e285688d9f3216fb67, c3cba8be0dc24b36a49c7d4b689a6888
I20260924 23:09:48.909137 17276 raft_consensus.cc:2726] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:09:49.622195 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:49.622003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:50241)
0924 23:09:49.622116 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:49.622150 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 23:09:49.668288 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:49.668083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:51471)
0924 23:09:49.668207 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:49.668248 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260924 23:09:49.813191 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:49.812986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:57699)
0924 23:09:49.813118 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:49.813154 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260924 23:09:50.622851 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:50.622636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:49839)
0924 23:09:50.622762 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:50.622804 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260924 23:09:50.668990 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:50.668796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:49463)
0924 23:09:50.668914 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:50.668951 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260924 23:09:50.813897 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:50.813651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:52477)
0924 23:09:50.813793 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:50.813834 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260924 23:09:51.060461 17286 raft_consensus.cc:482] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:09:51.060571 17286 raft_consensus.cc:504] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:51.060779 17286 leader_election.cc:288] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053), c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029)
W20260924 23:09:51.061195 16890 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
W20260924 23:09:51.061343 16891 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:51.061408 16891 leader_election.cc:302] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055; no voters: 79ecfb313e6245e285688d9f3216fb67, c3cba8be0dc24b36a49c7d4b689a6888
I20260924 23:09:51.061527 17286 raft_consensus.cc:2726] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:09:51.623570 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:51.623291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:34681)
0924 23:09:51.623443 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:51.623489 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260924 23:09:51.669692 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:51.669435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:50835)
0924 23:09:51.669589 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:51.669626 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260924 23:09:51.814584 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:51.814354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:56409)
0924 23:09:51.814489 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:51.814540 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260924 23:09:52.624416 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:52.624198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:47205)
0924 23:09:52.624331 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:52.624370 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260924 23:09:52.670331 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:52.670152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:50313)
0924 23:09:52.670262 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:52.670293 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 23:09:52.815280 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:52.815051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:59823)
0924 23:09:52.815204 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:52.815239 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 23:09:53.625218 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:53.624987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:54715)
0924 23:09:53.625127 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:53.625170 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260924 23:09:53.670964 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:53.670746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:38251)
0924 23:09:53.670883 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:53.670922 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":55,"threads_started":1}
I20260924 23:09:53.815977 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:53.815744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:33183)
0924 23:09:53.815880 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:53.815936 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260924 23:09:53.879249 17296 raft_consensus.cc:482] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:09:53.879383 17296 raft_consensus.cc:504] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:53.879596 17296 leader_election.cc:288] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053), c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029)
W20260924 23:09:53.879946 16891 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
W20260924 23:09:53.880112 16890 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:53.880200 16890 leader_election.cc:302] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055; no voters: 79ecfb313e6245e285688d9f3216fb67, c3cba8be0dc24b36a49c7d4b689a6888
I20260924 23:09:53.880296 17296 raft_consensus.cc:2726] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:09:54.626082 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:54.625727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:48519)
0924 23:09:54.625993 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:54.626034 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":79,"threads_started":1}
I20260924 23:09:54.671828 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:54.671591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:45381)
0924 23:09:54.671748 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:54.671785 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260924 23:09:54.816723 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:54.816491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:55019)
0924 23:09:54.816643 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:54.816682 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260924 23:09:55.626856 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:55.626633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:58653)
0924 23:09:55.626768 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:55.626809 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260924 23:09:55.672492 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:55.672251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:36357)
0924 23:09:55.672405 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:55.672447 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260924 23:09:55.817444 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:55.817223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:42715)
0924 23:09:55.817359 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:55.817401 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260924 23:09:56.627614 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:56.627381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:51769)
0924 23:09:56.627529 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:56.627568 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260924 23:09:56.673198 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:56.672953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:34887)
0924 23:09:56.673113 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:56.673154 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260924 23:09:56.818264 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:56.818020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:54245)
0924 23:09:56.818177 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:56.818220 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
I20260924 23:09:57.628419 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:57.628156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:34711)
0924 23:09:57.628307 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:57.628348 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":66,"threads_started":1}
I20260924 23:09:57.674322 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:57.673855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:33061)
0924 23:09:57.674065 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:57.674111 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":121,"threads_started":1}
I20260924 23:09:57.819002 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:57.818770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:51319)
0924 23:09:57.818913 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:57.818958 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260924 23:09:58.629171 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:58.628951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:51713)
0924 23:09:58.629086 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:58.629125 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260924 23:09:58.675032 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:58.674811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:48957)
0924 23:09:58.674949 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:58.674989 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260924 23:09:58.819725 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:58.819505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:39791)
0924 23:09:58.819647 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:58.819680 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260924 23:09:58.831087 17312 raft_consensus.cc:482] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:09:58.831183 17312 raft_consensus.cc:504] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79ecfb313e6245e285688d9f3216fb67" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34053 } } peers { permanent_uuid: "c3cba8be0dc24b36a49c7d4b689a6888" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39029 } } peers { permanent_uuid: "127bc56fcfc247f18eb1f4612d2c2055" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33589 } }
I20260924 23:09:58.831382 17312 leader_election.cc:288] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053), c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029)
W20260924 23:09:58.831787 16890 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3cba8be0dc24b36a49c7d4b689a6888 (127.0.98.3:39029): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
W20260924 23:09:58.831928 16891 leader_election.cc:341] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79ecfb313e6245e285688d9f3216fb67 (127.0.98.2:34053): Not found: Tablet not found: 2c40009a2f7e401aa7c36a591834b907
I20260924 23:09:58.832002 16891 leader_election.cc:302] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 127bc56fcfc247f18eb1f4612d2c2055; no voters: 79ecfb313e6245e285688d9f3216fb67, c3cba8be0dc24b36a49c7d4b689a6888
I20260924 23:09:58.832080 17312 raft_consensus.cc:2726] T 2c40009a2f7e401aa7c36a591834b907 P 127bc56fcfc247f18eb1f4612d2c2055 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 23:09:59.629966 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:59.629666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:51325)
0924 23:09:59.629826 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:59.629864 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260924 23:09:59.675725 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:59.675490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:36093)
0924 23:09:59.675645 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:59.675681 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260924 23:09:59.820405 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:09:59.820204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:38307)
0924 23:09:59.820317 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 23:09:59.820353 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260924 23:10:00.630738 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:00.630507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:41899)
0924 23:10:00.630651 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:00.630691 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260924 23:10:00.676716 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:00.676385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:42263)
0924 23:10:00.676603 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:00.676670 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260924 23:10:00.821197 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:00.820978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:33823)
0924 23:10:00.821112 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:00.821154 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 23:10:01.631446 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:01.631229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:57897)
0924 23:10:01.631364 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:01.631400 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260924 23:10:01.677467 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:01.677238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:50781)
0924 23:10:01.677377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:01.677418 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260924 23:10:01.822005 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:01.821709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:39891)
0924 23:10:01.821869 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:01.821943 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":120,"threads_started":1}
I20260924 23:10:02.632319 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:02.632085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.1:52033)
0924 23:10:02.632227 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:02.632271 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260924 23:10:02.678258 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:02.678027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.2:55319)
0924 23:10:02.678171 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:02.678214 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260924 23:10:02.822899 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:02.822663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43791 (local address 127.0.98.3:53811)
0924 23:10:02.822802 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:02.822855 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260924 23:10:02.826536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 16839
I20260924 23:10:02.831310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17007
I20260924 23:10:02.836292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17140
2026-09-24T23:10:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58655 ms)
[----------] 30 tests from SecurityITest (233397 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:02 dist-test-slave-ph9j krb5kdc[17327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:02 dist-test-slave-ph9j krb5kdc[17327](info): set up 2 sockets
Sep 24 23:10:02 dist-test-slave-ph9j krb5kdc[17327](info): commencing operation
krb5kdc: starting...
W20260924 23:10:03.865603 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:03 dist-test-slave-ph9j krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:03.983035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:04.062687 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:04.062947 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:04.062986 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:04.064167 17340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:04.064245 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:04.064273 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:04.064306 17340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:04.064329 17340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:04.065768 17340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:04.066075 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:04.066308 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:04.071758 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.071820 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.071835 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:04.071830 17340 server_base.cc:1034] running on GCE node
I20260924 23:10:04.072230 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:04.072535 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:04.073704 17340 hybrid_clock.cc:648] HybridClock initialized: now 1790291404073706 us; error 45 us; skew 500 ppm
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:04.081063 17340 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:04.081481 17340 webserver.cc:460] Webserver started at http://127.0.98.62:44125/ using document root <none> and password file <none>
I20260924 23:10:04.081672 17340 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:04.081713 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:04.081802 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:04.082916 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "c8f77d5e0ee2418489698fcd639ae88f"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "38ff062f22bb38c7918156c749ea659a"
server_key_iv: "b055d27f362960f3c40046e9685ba269"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.083210 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "c8f77d5e0ee2418489698fcd639ae88f"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "38ff062f22bb38c7918156c749ea659a"
server_key_iv: "b055d27f362960f3c40046e9685ba269"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.084465 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:10:04.085150 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.085351 17340 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:04.085441 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "c8f77d5e0ee2418489698fcd639ae88f"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "38ff062f22bb38c7918156c749ea659a"
server_key_iv: "b055d27f362960f3c40046e9685ba269"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.085500 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:04.096558 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:04.098897 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:04.099007 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:04.102802 17340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36379
I20260924 23:10:04.102833 17407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36379 every 8 connection(s)
I20260924 23:10:04.103232 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:04.104089 17408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:04.106413 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f: Bootstrap starting.
I20260924 23:10:04.106983 17408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:04.107271 17408 log.cc:824] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:04.107980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17340
I20260924 23:10:04.108047 17408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f: No bootstrap required, opened a new log
I20260924 23:10:04.108105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:04.108316 392 external_mini_cluster.cc:1428] Setting key 12d52c05089112edbbab7ced63c04fb0
I20260924 23:10:04.110091 17408 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f77d5e0ee2418489698fcd639ae88f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 23:10:04.110261 17408 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:04.110301 17408 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f77d5e0ee2418489698fcd639ae88f, State: Initialized, Role: FOLLOWER
I20260924 23:10:04.110476 17408 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f77d5e0ee2418489698fcd639ae88f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 23:10:04.110570 17408 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:04.110610 17408 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:04.110659 17408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:04.111677 17408 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f77d5e0ee2418489698fcd639ae88f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 23:10:04.111764 17408 leader_election.cc:302] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8f77d5e0ee2418489698fcd639ae88f; no voters:
I20260924 23:10:04.111964 17408 leader_election.cc:288] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:04.111985 17413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:04.112104 17413 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [term 1 LEADER]: Becoming Leader. State: Replica: c8f77d5e0ee2418489698fcd639ae88f, State: Running, Role: LEADER
I20260924 23:10:04.112188 17413 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f77d5e0ee2418489698fcd639ae88f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } }
I20260924 23:10:04.112241 17408 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:04.112504 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8f77d5e0ee2418489698fcd639ae88f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f77d5e0ee2418489698fcd639ae88f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260924 23:10:04.112607 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:04.112769 17413 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8f77d5e0ee2418489698fcd639ae88f. Latest consensus state: current_term: 1 leader_uuid: "c8f77d5e0ee2418489698fcd639ae88f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8f77d5e0ee2418489698fcd639ae88f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36379 } } }
I20260924 23:10:04.112856 17413 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:04.113201 17418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:04.113719 17418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:04.115334 17418 catalog_manager.cc:1347] Generated new cluster ID: 73b23acd8cce4ccc8f8b54dd2c8971e5
I20260924 23:10:04.115386 17418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:04.120191 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.108876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.0.98.62:36379)
0924 23:10:04.109034 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.109042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.109065 (+ 23us) server_negotiation.cc:407] Connection header received
0924 23:10:04.110069 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.110078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.110129 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.110250 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:04.111108 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.112239 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.113430 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.114106 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.114228 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.116854 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:04.116858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:04.116859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:04.116871 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:04.118604 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.119000 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.119006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.119006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.119125 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.119342 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.119343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.119343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.119596 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:04.119781 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.119847 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.119992 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260924 23:10:04.128104 17418 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:04.128692 17418 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:04.136191 17418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f: Generated new TSK 0
I20260924 23:10:04.136409 17418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:10:04.168748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:04.270404 17434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:04.270607 17434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:04.270632 17434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:04.271804 17434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:04.271844 17434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:04.271871 17434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:10:04.273370 17434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:04.273605 17434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:04.273926 17434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:04.278883 17442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.279062 17440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.279063 17439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:04.279189 17434 server_base.cc:1034] running on GCE node
I20260924 23:10:04.279448 17434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:04.279665 17434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:04.280812 17434 hybrid_clock.cc:648] HybridClock initialized: now 1790291404280794 us; error 26 us; skew 500 ppm
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:04.283454 17434 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:10:04.283840 17434 webserver.cc:460] Webserver started at http://127.0.98.1:40157/ using document root <none> and password file <none>
I20260924 23:10:04.284044 17434 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:04.284085 17434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:04.284176 17434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:04.285151 17434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "577353375e0e444fbec31f428424509f"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "9077841301f36d135f300cfff0b5a089"
server_key_iv: "c0e6d02d4c494b83ccfbc96f5da76b6e"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.285452 17434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "577353375e0e444fbec31f428424509f"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "9077841301f36d135f300cfff0b5a089"
server_key_iv: "c0e6d02d4c494b83ccfbc96f5da76b6e"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.286777 17434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:04.287468 17449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.287635 17434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:04.287739 17434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "577353375e0e444fbec31f428424509f"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "9077841301f36d135f300cfff0b5a089"
server_key_iv: "c0e6d02d4c494b83ccfbc96f5da76b6e"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.287813 17434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:04.307818 17434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:04.310349 17434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:04.310520 17434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:04.310768 17434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:04.311115 17434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:04.311172 17434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.311215 17434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:04.311244 17434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.317373 17434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39659
I20260924 23:10:04.317420 17562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39659 every 8 connection(s)
I20260924 23:10:04.317754 17434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:04.324698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17434
I20260924 23:10:04.324818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:10:04.325069 392 external_mini_cluster.cc:1428] Setting key ba5dae392bd94739751a26d5da9f8aa3
I20260924 23:10:04.327450 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.318471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39557 (local address 127.0.98.62:36379)
0924 23:10:04.318541 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.318543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.318730 (+ 187us) server_negotiation.cc:407] Connection header received
0924 23:10:04.319639 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.319640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.319653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.319715 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:04.320315 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.320817 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.321787 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.322077 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.322133 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.324379 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:04.324380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:04.324380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:04.324388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:04.325912 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.326390 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.326391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.326391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.326441 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.326717 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.326718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.326718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.326816 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:04.326864 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.327193 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.327306 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 23:10:04.327653 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.318446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.1:39557)
0924 23:10:04.318599 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.318640 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.319539 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.319770 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.319773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.320023 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:04.320219 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.320222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.320897 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.320898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.321710 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.321712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.322184 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.322185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.322300 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.322875 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:04.322884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:04.324282 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:04.325973 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:04.325980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:04.325983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:04.326306 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:04.326503 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:04.326504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:04.326505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:04.326654 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:04.326878 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:04.326884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:04.327060 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.327174 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.327366 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 23:10:04.328127 17563 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 23:10:04.328234 17563 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:04.328449 17563 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 23:10:04.328930 17372 ts_manager.cc:194] Registered new tserver with Master: 577353375e0e444fbec31f428424509f (127.0.98.1:39659)
I20260924 23:10:04.329569 17372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39557
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:10:04.350895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:04.432657 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:04.432863 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:04.432888 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:04.434150 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:04.434193 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:04.434221 17568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:10:04.435714 17568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:04.435957 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:04.436188 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:04.441236 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:04.441394 17568 server_base.cc:1034] running on GCE node
W20260924 23:10:04.441255 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.441264 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:04.441723 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:04.442010 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:04.443176 17568 hybrid_clock.cc:648] HybridClock initialized: now 1790291404443161 us; error 45 us; skew 500 ppm
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:04.445676 17568 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:10:04.446185 17568 webserver.cc:460] Webserver started at http://127.0.98.2:37327/ using document root <none> and password file <none>
I20260924 23:10:04.446419 17568 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:04.446486 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:04.446643 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:04.447675 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "afbc0c8a65a44b74ba77106813f1b841"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "922f91769f4498d93645665e47da8c3d"
server_key_iv: "196e9e9fbf9684e926c8e558593d001f"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.448029 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "afbc0c8a65a44b74ba77106813f1b841"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "922f91769f4498d93645665e47da8c3d"
server_key_iv: "196e9e9fbf9684e926c8e558593d001f"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.449335 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:04.450084 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.450242 17568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:04.450349 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "afbc0c8a65a44b74ba77106813f1b841"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "922f91769f4498d93645665e47da8c3d"
server_key_iv: "196e9e9fbf9684e926c8e558593d001f"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.450423 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:04.461071 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:04.463016 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:04.463182 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:04.463414 17568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:04.463745 17568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:04.463800 17568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.463843 17568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:04.463874 17568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.469662 17568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34127
I20260924 23:10:04.469743 17696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34127 every 8 connection(s)
I20260924 23:10:04.470103 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:10:04.475515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17568
I20260924 23:10:04.475659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:10:04.475872 392 external_mini_cluster.cc:1428] Setting key b805bb5cb56eb2f31c6f4c746df0a617
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:04.480052 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.470882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55367 (local address 127.0.98.62:36379)
0924 23:10:04.471000 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.471002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.471215 (+ 213us) server_negotiation.cc:407] Connection header received
0924 23:10:04.472141 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.472143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.472154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.472211 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:04.472837 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.473370 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.474520 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.474713 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.474775 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.477105 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:04.477107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:04.477107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:04.477116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:04.478599 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.479107 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.479108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.479108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.479158 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.479432 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.479433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.479433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.479532 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:04.479582 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.479913 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.479987 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 23:10:04.480351 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.470856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.2:55367)
0924 23:10:04.471085 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.471122 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.472002 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.472286 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.472289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.472552 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:04.472742 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.472746 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.473453 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.473453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.474422 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.474424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.474790 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.474791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.474887 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.475430 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:04.475440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:04.476991 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:04.478680 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:04.478687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:04.478690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:04.479019 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:04.479222 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:04.479223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:04.479223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:04.479370 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:04.479593 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:04.479599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:04.479768 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.479884 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.480073 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":113,"threads_started":1}
I20260924 23:10:04.480836 17697 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 23:10:04.480957 17697 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:04.481159 17697 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 23:10:04.481608 17372 ts_manager.cc:194] Registered new tserver with Master: afbc0c8a65a44b74ba77106813f1b841 (127.0.98.2:34127)
I20260924 23:10:04.482209 17372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55367
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:10:04.501124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:04.582856 17702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:04.583070 17702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:04.583094 17702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:04.584165 17702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:04.584198 17702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:04.584223 17702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:10:04.585644 17702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:04.585822 17702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:04.586014 17702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:04.590924 17710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.590929 17708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.590955 17707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:04.591195 17702 server_base.cc:1034] running on GCE node
I20260924 23:10:04.591440 17702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:04.591681 17702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:04.592839 17702 hybrid_clock.cc:648] HybridClock initialized: now 1790291404592821 us; error 40 us; skew 500 ppm
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:04.595482 17702 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:10:04.595939 17702 webserver.cc:460] Webserver started at http://127.0.98.3:42949/ using document root <none> and password file <none>
I20260924 23:10:04.596166 17702 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:04.596231 17702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:04.596374 17702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:04.597402 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a2a67cd30054a158eee3c1aa4a691fc"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "44fd2ac293d45f906d8259ecf44b2e29"
server_key_iv: "aa5ab63e8749231a74f95743d59204af"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.597770 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a2a67cd30054a158eee3c1aa4a691fc"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "44fd2ac293d45f906d8259ecf44b2e29"
server_key_iv: "aa5ab63e8749231a74f95743d59204af"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.599107 17702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:04.599836 17717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.600046 17702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:04.600160 17702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "4a2a67cd30054a158eee3c1aa4a691fc"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "44fd2ac293d45f906d8259ecf44b2e29"
server_key_iv: "aa5ab63e8749231a74f95743d59204af"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.600235 17702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:04.613965 17702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:04.616183 17702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:04.616344 17702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:04.616578 17702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:04.616889 17702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:04.616931 17702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.617002 17702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:04.617031 17702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.622951 17702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33769
I20260924 23:10:04.623059 17830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33769 every 8 connection(s)
I20260924 23:10:04.623365 17702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:10:04.625708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17702
I20260924 23:10:04.625814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:10:04.626034 392 external_mini_cluster.cc:1428] Setting key 6ed700e8b9fe75ba47a873c6de610403
Sep 24 23:10:04 dist-test-slave-ph9j krb5kdc[17327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:04.632611 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.624103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38353 (local address 127.0.98.62:36379)
0924 23:10:04.624177 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.624179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.624364 (+ 185us) server_negotiation.cc:407] Connection header received
0924 23:10:04.625284 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.625285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.625296 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.625357 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:04.626261 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.627020 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.627989 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.628177 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.628235 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.630265 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:04.630266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:04.630267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:04.630273 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:04.631450 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.631794 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.631795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.631795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.631841 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.632039 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.632039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.632039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.632141 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:04.632212 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.632485 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.632548 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 23:10:04.632861 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.624070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36379 (local address 127.0.98.3:38353)
0924 23:10:04.624231 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.624278 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.625187 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.625410 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.625414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.625805 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:04.626135 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.626141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.627108 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.627109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.627913 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.627915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.628250 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.628251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.628346 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.628798 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:04.628807 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:04.630187 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:04.631515 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:04.631521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:04.631523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:04.631732 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:04.631895 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:04.631896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:04.631896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:04.631986 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:04.632204 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:04.632208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:04.632317 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.632468 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.632597 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 23:10:04.633343 17831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36379
I20260924 23:10:04.633452 17831 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:04.633648 17831 heartbeater.cc:511] Master 127.0.98.62:36379 requested a full tablet report, sending...
I20260924 23:10:04.634061 17372 ts_manager.cc:194] Registered new tserver with Master: 4a2a67cd30054a158eee3c1aa4a691fc (127.0.98.3:33769)
I20260924 23:10:04.634421 17372 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38353
I20260924 23:10:04.637503 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:10:04.644343 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.638790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.0.98.62:36379)
0924 23:10:04.638859 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.638861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.638898 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:10:04.638948 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.638948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.638956 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.639005 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:04.639502 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.639960 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.640654 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.640821 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.640867 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.641994 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:04.641995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:04.641995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:04.642001 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:04.643344 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.643636 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.643637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.643637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.643685 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:04.643910 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:04.643911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:04.643911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:04.644032 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:04.644102 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.644234 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.644287 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:10:04.645829 17372 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37444:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:10:04.647454 17372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:10:04.653990 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.649715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34127 (local address 127.0.0.1:33122)
0924 23:10:04.649860 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.649932 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.649995 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.650591 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.650592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.650594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:04.650651 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.650653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.651528 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.651529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.652733 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.652735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.653661 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.653662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.653738 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.653754 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.653821 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.653914 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
I20260924 23:10:04.654093 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.649716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:36390)
0924 23:10:04.649862 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.649932 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.649994 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.650451 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.650455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.650457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:04.650528 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.650530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.651885 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.651886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.652910 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.652912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.653774 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.653775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.653918 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.653923 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.653977 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.654021 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260924 23:10:04.654120 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.649843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33122 (local address 127.0.98.2:34127)
0924 23:10:04.650402 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.650407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.650417 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:04.650460 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.650462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.650478 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.650728 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:04.650790 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.651429 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.652813 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.653573 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.653650 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.653712 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.653954 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.654032 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":113,"threads_started":1}
I20260924 23:10:04.654325 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.650087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.98.1:39659)
0924 23:10:04.650258 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.650263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.650274 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:10:04.650314 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.650317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.650333 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.650630 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:04.650716 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.651775 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.652975 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.653708 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.653771 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.653821 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.654194 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.654247 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260924 23:10:04.655066 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.650021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33769 (local address 127.0.0.1:32954)
0924 23:10:04.650129 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.650137 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.650177 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.650535 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.650536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.650538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:04.650577 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.650579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.652988 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.652988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.653761 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.653763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.654802 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.654802 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.654868 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.654872 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.654965 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.655004 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":76,"threads_started":1}
I20260924 23:10:04.654959 17631 tablet_service.cc:1467] Processing CreateTablet for tablet 25d656eb01464fa2a62d5cc28a550aa5 (DEFAULT_TABLE table=test-table [id=9a6caa3ae24d4c3a93d05b19e56e0b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:04.655112 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.650180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.0.98.3:33769)
0924 23:10:04.650329 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.650333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.650342 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:04.650376 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.650378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.650392 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.650461 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:04.650655 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.651775 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.653873 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.654649 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.654709 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.654769 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.654983 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.655024 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260924 23:10:04.655309 17631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d656eb01464fa2a62d5cc28a550aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:04.655231 17497 tablet_service.cc:1467] Processing CreateTablet for tablet 25d656eb01464fa2a62d5cc28a550aa5 (DEFAULT_TABLE table=test-table [id=9a6caa3ae24d4c3a93d05b19e56e0b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:04.655508 17497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d656eb01464fa2a62d5cc28a550aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:04.655830 17765 tablet_service.cc:1467] Processing CreateTablet for tablet 25d656eb01464fa2a62d5cc28a550aa5 (DEFAULT_TABLE table=test-table [id=9a6caa3ae24d4c3a93d05b19e56e0b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:04.656121 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d656eb01464fa2a62d5cc28a550aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:04.658552 17848 tablet_bootstrap.cc:492] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Bootstrap starting.
I20260924 23:10:04.658979 17850 tablet_bootstrap.cc:492] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f: Bootstrap starting.
I20260924 23:10:04.659089 17849 tablet_bootstrap.cc:492] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Bootstrap starting.
I20260924 23:10:04.659332 17848 tablet_bootstrap.cc:654] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:04.659628 17850 tablet_bootstrap.cc:654] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:04.659654 17849 tablet_bootstrap.cc:654] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:04.659890 17848 log.cc:824] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:04.660259 17849 log.cc:824] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:04.660250 17850 log.cc:824] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:04.660658 17848 tablet_bootstrap.cc:492] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: No bootstrap required, opened a new log
I20260924 23:10:04.660737 17848 ts_tablet_manager.cc:1397] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:04.661038 17849 tablet_bootstrap.cc:492] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: No bootstrap required, opened a new log
I20260924 23:10:04.661128 17849 ts_tablet_manager.cc:1397] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:04.661336 17850 tablet_bootstrap.cc:492] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f: No bootstrap required, opened a new log
I20260924 23:10:04.661438 17850 ts_tablet_manager.cc:1397] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:10:04.662401 17848 raft_consensus.cc:348] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.662564 17848 raft_consensus.cc:374] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:04.662618 17848 raft_consensus.cc:729] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2a67cd30054a158eee3c1aa4a691fc, State: Initialized, Role: FOLLOWER
I20260924 23:10:04.662734 17848 consensus_queue.cc:260] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.663008 17848 ts_tablet_manager.cc:1428] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:10:04.663152 17831 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260924 23:10:04.663188 17849 raft_consensus.cc:348] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.663363 17849 raft_consensus.cc:374] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:04.663406 17849 raft_consensus.cc:729] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbc0c8a65a44b74ba77106813f1b841, State: Initialized, Role: FOLLOWER
I20260924 23:10:04.663547 17849 consensus_queue.cc:260] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.663491 17850 raft_consensus.cc:348] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.663651 17850 raft_consensus.cc:374] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:04.663697 17850 raft_consensus.cc:729] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577353375e0e444fbec31f428424509f, State: Initialized, Role: FOLLOWER
I20260924 23:10:04.663815 17849 ts_tablet_manager.cc:1428] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:10:04.663815 17850 consensus_queue.cc:260] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.663925 17697 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
I20260924 23:10:04.664068 17850 ts_tablet_manager.cc:1428] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:10:04.664175 17563 heartbeater.cc:503] Master 127.0.98.62:36379 was elected leader, sending a full tablet report...
W20260924 23:10:04.720733 17698 tablet.cc:2369] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:04.751022 17855 raft_consensus.cc:482] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:10:04.751223 17855 raft_consensus.cc:504] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.751623 17855 leader_election.cc:288] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a2a67cd30054a158eee3c1aa4a691fc (127.0.98.3:33769), 577353375e0e444fbec31f428424509f (127.0.98.1:39659)
I20260924 23:10:04.754865 17699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.751850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33769 (local address 127.0.98.2:45411)
0924 23:10:04.751924 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.751939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.752001 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.752152 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.752153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.752155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:04.752254 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.752256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.753027 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.753028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.753869 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.753926 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.754598 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.754599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.754683 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.754687 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.754738 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.754772 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 23:10:04.754882 17847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.751965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45411 (local address 127.0.98.3:33769)
0924 23:10:04.752010 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.752012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.752043 (+ 31us) server_negotiation.cc:407] Connection header received
0924 23:10:04.752072 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.752073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.752082 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.752128 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:04.752347 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.752941 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.754031 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.754524 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.754585 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.754594 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.754769 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.754816 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:10:04.754948 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.751985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35683 (local address 127.0.98.1:39659)
0924 23:10:04.752033 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.752035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.752209 (+ 174us) server_negotiation.cc:407] Connection header received
0924 23:10:04.752244 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.752245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.752258 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.752317 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:04.752511 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.753081 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.754142 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.754576 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.754646 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.754655 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.754836 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.754877 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:10:04.754976 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.751850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.98.2:35683)
0924 23:10:04.752128 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.752138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:04.752189 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:04.752337 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:04.752338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:04.752339 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:04.752415 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.752417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.753181 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.753182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.754030 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:04.754032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.754659 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.754660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:04.754757 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.754763 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:10:04.754819 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 23:10:04.754856 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":74,"thread_start_us":126,"threads_started":1}
I20260924 23:10:04.755236 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d656eb01464fa2a62d5cc28a550aa5" candidate_uuid: "afbc0c8a65a44b74ba77106813f1b841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" is_pre_election: true
I20260924 23:10:04.755398 17785 raft_consensus.cc:2454] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbc0c8a65a44b74ba77106813f1b841 in term 0.
I20260924 23:10:04.755693 17585 leader_election.cc:302] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a2a67cd30054a158eee3c1aa4a691fc, afbc0c8a65a44b74ba77106813f1b841; no voters:
I20260924 23:10:04.755717 17517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d656eb01464fa2a62d5cc28a550aa5" candidate_uuid: "afbc0c8a65a44b74ba77106813f1b841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577353375e0e444fbec31f428424509f" is_pre_election: true
I20260924 23:10:04.755847 17517 raft_consensus.cc:2454] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbc0c8a65a44b74ba77106813f1b841 in term 0.
I20260924 23:10:04.755832 17855 raft_consensus.cc:2781] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:10:04.755897 17855 raft_consensus.cc:482] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:10:04.755924 17855 raft_consensus.cc:3037] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:04.756614 17855 raft_consensus.cc:504] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.756752 17855 leader_election.cc:288] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [CANDIDATE]: Term 1 election: Requested vote from peers 4a2a67cd30054a158eee3c1aa4a691fc (127.0.98.3:33769), 577353375e0e444fbec31f428424509f (127.0.98.1:39659)
I20260924 23:10:04.756870 17517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d656eb01464fa2a62d5cc28a550aa5" candidate_uuid: "afbc0c8a65a44b74ba77106813f1b841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577353375e0e444fbec31f428424509f"
I20260924 23:10:04.756942 17517 raft_consensus.cc:3037] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:04.757555 17517 raft_consensus.cc:2454] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbc0c8a65a44b74ba77106813f1b841 in term 1.
I20260924 23:10:04.757678 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d656eb01464fa2a62d5cc28a550aa5" candidate_uuid: "afbc0c8a65a44b74ba77106813f1b841" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a2a67cd30054a158eee3c1aa4a691fc"
I20260924 23:10:04.757738 17587 leader_election.cc:302] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577353375e0e444fbec31f428424509f, afbc0c8a65a44b74ba77106813f1b841; no voters:
I20260924 23:10:04.757761 17785 raft_consensus.cc:3037] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:04.757922 17855 raft_consensus.cc:2781] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:04.758095 17855 raft_consensus.cc:686] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 1 LEADER]: Becoming Leader. State: Replica: afbc0c8a65a44b74ba77106813f1b841, State: Running, Role: LEADER
I20260924 23:10:04.758212 17855 consensus_queue.cc:237] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } }
I20260924 23:10:04.758657 17785 raft_consensus.cc:2454] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbc0c8a65a44b74ba77106813f1b841 in term 1.
I20260924 23:10:04.759215 17372 catalog_manager.cc:5495] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 reported cstate change: term changed from 0 to 1, leader changed from <none> to afbc0c8a65a44b74ba77106813f1b841 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "afbc0c8a65a44b74ba77106813f1b841" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afbc0c8a65a44b74ba77106813f1b841" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 } health_report { overall_health: UNKNOWN } } }
I20260924 23:10:04.768687 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.766579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.0.98.2:34127)
0924 23:10:04.766619 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.766621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.766630 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:04.766682 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.766683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.766699 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.766758 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:04.766880 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.767330 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.768048 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.768235 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.768285 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.768419 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:04.768509 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:04.768569 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.768601 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.768617 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:10:04.770387 17517 raft_consensus.cc:1261] T 25d656eb01464fa2a62d5cc28a550aa5 P 577353375e0e444fbec31f428424509f [term 1 FOLLOWER]: Refusing update from remote peer afbc0c8a65a44b74ba77106813f1b841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:04.770409 17785 raft_consensus.cc:1261] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 1 FOLLOWER]: Refusing update from remote peer afbc0c8a65a44b74ba77106813f1b841: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:04.770735 17858 consensus_queue.cc:1035] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "577353375e0e444fbec31f428424509f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:04.770862 17855 consensus_queue.cc:1035] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a2a67cd30054a158eee3c1aa4a691fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:04.773062 17861 mvcc.cc:205] Tried to move back new op lower bound from 7333033593936662528 to 7333033593890205696. Current Snapshot: MvccSnapshot[applied={T|T < 7333033593936662528}]
I20260924 23:10:04.773767 17863 mvcc.cc:205] Tried to move back new op lower bound from 7333033593936662528 to 7333033593890205696. Current Snapshot: MvccSnapshot[applied={T|T < 7333033593936662528}]
I20260924 23:10:04.775699 17372 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:04.775779 17372 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37444:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:04.776552 17372 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8f77d5e0ee2418489698fcd639ae88f: Sending DeleteTablet for 3 replicas of tablet 25d656eb01464fa2a62d5cc28a550aa5
I20260924 23:10:04.776893 17765 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d656eb01464fa2a62d5cc28a550aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32954
I20260924 23:10:04.776924 17631 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d656eb01464fa2a62d5cc28a550aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33122
I20260924 23:10:04.776899 17497 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d656eb01464fa2a62d5cc28a550aa5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36390
I20260924 23:10:04.777174 17870 tablet_replica.cc:330] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: stopping tablet replica
I20260924 23:10:04.777311 17870 raft_consensus.cc:2229] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:04.777395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17434
I20260924 23:10:04.777472 17870 raft_consensus.cc:2258] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:04.778940 17870 ts_tablet_manager.cc:1905] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:04.779429 17872 tablet_replica.cc:330] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: stopping tablet replica
I20260924 23:10:04.779526 17872 raft_consensus.cc:2229] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:10:04.779646 17872 raft_consensus.cc:2258] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:04.780237 17872 ts_tablet_manager.cc:1905] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:04.780696 17870 ts_tablet_manager.cc:1918] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:04.780771 17870 log.cc:1196] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/25d656eb01464fa2a62d5cc28a550aa5
I20260924 23:10:04.781044 17870 ts_tablet_manager.cc:1939] T 25d656eb01464fa2a62d5cc28a550aa5 P 4a2a67cd30054a158eee3c1aa4a691fc: Deleting consensus metadata
I20260924 23:10:04.781528 17357 catalog_manager.cc:4841] TS 4a2a67cd30054a158eee3c1aa4a691fc (127.0.98.3:33769): tablet 25d656eb01464fa2a62d5cc28a550aa5 (table test-table [id=9a6caa3ae24d4c3a93d05b19e56e0b2e]) successfully deleted
I20260924 23:10:04.781795 17872 ts_tablet_manager.cc:1918] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:04.781868 17872 log.cc:1196] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/25d656eb01464fa2a62d5cc28a550aa5
I20260924 23:10:04.782161 17872 ts_tablet_manager.cc:1939] T 25d656eb01464fa2a62d5cc28a550aa5 P afbc0c8a65a44b74ba77106813f1b841: Deleting consensus metadata
I20260924 23:10:04.782532 17357 catalog_manager.cc:4841] TS afbc0c8a65a44b74ba77106813f1b841 (127.0.98.2:34127): tablet 25d656eb01464fa2a62d5cc28a550aa5 (table test-table [id=9a6caa3ae24d4c3a93d05b19e56e0b2e]) successfully deleted
W20260924 23:10:04.785005 17359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39659 (error 108)
I20260924 23:10:04.785216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17568
I20260924 23:10:04.786027 17845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.785851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39659 (local address 127.0.0.1:36404)
0924 23:10:04.785946 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:04.785988 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 23:10:04.786185 17359 catalog_manager.cc:4584] TS 577353375e0e444fbec31f428424509f (127.0.98.1:39659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25d656eb01464fa2a62d5cc28a550aa5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39659: connect: Connection refused (error 111)
I20260924 23:10:04.790627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17702
I20260924 23:10:04.795347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17340
2026-09-24T23:10:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1959 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T23:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:04Z Disabled control of system clock
I20260924 23:10:04.823565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34109
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:04.904372 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:04.904577 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:04.904597 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:04.905663 17877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:04.905699 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:04.905711 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:04.905723 17877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:04.905735 17877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:04.906989 17877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:04.907168 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:04.907397 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:10:04.912163 17877 server_base.cc:1034] running on GCE node
W20260924 23:10:04.912133 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.912135 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:04.912169 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:04.912669 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:04.912907 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:04.914048 17877 hybrid_clock.cc:648] HybridClock initialized: now 1790291404914037 us; error 26 us; skew 500 ppm
I20260924 23:10:04.915247 17877 webserver.cc:460] Webserver started at http://127.0.98.62:40253/ using document root <none> and password file <none>
I20260924 23:10:04.915449 17877 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:04.915489 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:04.915592 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:04.916548 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "26a5f11a71ac45a6878c1b839b201115"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "6fb331be3715ac4629f8150c0b9da7cf"
server_key_iv: "b7d5e92e5cf8163c8868754b26b78074"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.916843 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "26a5f11a71ac45a6878c1b839b201115"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "6fb331be3715ac4629f8150c0b9da7cf"
server_key_iv: "b7d5e92e5cf8163c8868754b26b78074"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.918120 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:10:04.918754 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.918911 17877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:04.918973 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "26a5f11a71ac45a6878c1b839b201115"
format_stamp: "Formatted at 2026-09-24 23:10:04 on dist-test-slave-ph9j"
server_key: "6fb331be3715ac4629f8150c0b9da7cf"
server_key_iv: "b7d5e92e5cf8163c8868754b26b78074"
server_key_version: "encryptionkey@0"
I20260924 23:10:04.919023 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:04.927304 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:04.927554 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:04.927662 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:04.931548 17877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34109
I20260924 23:10:04.931578 17943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34109 every 8 connection(s)
I20260924 23:10:04.931913 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:04.932824 17944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:04.935032 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: Bootstrap starting.
I20260924 23:10:04.935529 17944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:04.935763 17944 log.cc:824] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:04.936352 17944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: No bootstrap required, opened a new log
I20260924 23:10:04.937832 17944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a5f11a71ac45a6878c1b839b201115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 23:10:04.938054 17944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:04.938084 17944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a5f11a71ac45a6878c1b839b201115, State: Initialized, Role: FOLLOWER
I20260924 23:10:04.938123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17877
I20260924 23:10:04.938222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:04.938236 17944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a5f11a71ac45a6878c1b839b201115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 23:10:04.938304 17944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:04.938337 17944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:04.938377 17944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:04.938401 392 external_mini_cluster.cc:1428] Setting key 45991b941d3f866c03d23f2621b78de5
I20260924 23:10:04.939043 17944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a5f11a71ac45a6878c1b839b201115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 23:10:04.939153 17944 leader_election.cc:302] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26a5f11a71ac45a6878c1b839b201115; no voters:
I20260924 23:10:04.939307 17944 leader_election.cc:288] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:04.939333 17949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:04.939461 17949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [term 1 LEADER]: Becoming Leader. State: Replica: 26a5f11a71ac45a6878c1b839b201115, State: Running, Role: LEADER
I20260924 23:10:04.939568 17944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:04.939610 17949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a5f11a71ac45a6878c1b839b201115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } }
I20260924 23:10:04.939924 17949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a5f11a71ac45a6878c1b839b201115. Latest consensus state: current_term: 1 leader_uuid: "26a5f11a71ac45a6878c1b839b201115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a5f11a71ac45a6878c1b839b201115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } } }
I20260924 23:10:04.940061 17949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:04.940037 17950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26a5f11a71ac45a6878c1b839b201115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26a5f11a71ac45a6878c1b839b201115" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34109 } } }
I20260924 23:10:04.940519 17950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115 [sys.catalog]: This master's current role is: LEADER
W20260924 23:10:04.941706 17965 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:04.941761 17965 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:04.941804 17956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:04.941973 17956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:04.943802 17956 catalog_manager.cc:1347] Generated new cluster ID: ac1749eb863540a69c8fb31b7dfade49
I20260924 23:10:04.943857 17956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:04.944497 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:04.938905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.0.98.62:34109)
0924 23:10:04.939119 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:04.939126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:04.939176 (+ 50us) server_negotiation.cc:407] Connection header received
0924 23:10:04.939287 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:04.939295 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:04.939368 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:04.939515 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:04.939853 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.941299 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.942367 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:04.942896 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:04.943048 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:04.943686 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:04.943690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:04.943690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:04.943897 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:04.943941 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:04.943956 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:04.944025 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:04.944148 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 23:10:04.944303 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":107,"threads_started":1}
I20260924 23:10:04.952134 17956 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:04.952621 17956 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:04.964421 17956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: Generated new TSK 0
I20260924 23:10:04.964617 17956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:04.972781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--builtin_ntp_servers=127.0.98.20:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:05.064217 17968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:05.064414 17968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:05.064435 17968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:05.065569 17968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:05.065611 17968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:05.065636 17968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:10:05.067024 17968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:05.067273 17968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:05.067572 17968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:05.073626 17974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:05.073618 17976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:05.073901 17968 server_base.cc:1034] running on GCE node
W20260924 23:10:05.073653 17973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:05.074239 17968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:05.074494 17968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:05.075644 17968 hybrid_clock.cc:648] HybridClock initialized: now 1790291405075627 us; error 34 us; skew 500 ppm
I20260924 23:10:05.076930 17968 webserver.cc:460] Webserver started at http://127.0.98.1:38499/ using document root <none> and password file <none>
I20260924 23:10:05.077188 17968 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:05.077261 17968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:05.077453 17968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:05.078645 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "95f452866929d1a427100af2f5ed86fe"
server_key_iv: "d5672010899f9423bb8a6b733989006f"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.079000 17968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "95f452866929d1a427100af2f5ed86fe"
server_key_iv: "d5672010899f9423bb8a6b733989006f"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.080410 17968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:05.081243 17982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.081410 17968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:05.081521 17968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "95f452866929d1a427100af2f5ed86fe"
server_key_iv: "d5672010899f9423bb8a6b733989006f"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.081609 17968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:05.093163 17968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:05.093465 17968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:05.093616 17968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:05.093837 17968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:05.094241 17968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:05.094277 17968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.094300 17968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:05.094347 17968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.100265 17968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38105
I20260924 23:10:05.100345 18095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38105 every 8 connection(s)
I20260924 23:10:05.100708 17968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:10:05.105683 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.101462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40313 (local address 127.0.98.62:34109)
0924 23:10:05.101537 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.101540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.101800 (+ 260us) server_negotiation.cc:407] Connection header received
0924 23:10:05.102335 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.102336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.102348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.102400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:05.103031 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.103540 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.104474 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.104667 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.104721 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.105453 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:05.105454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:05.105454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:05.105458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:05.105471 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:05.105475 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:05.105517 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.105601 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.105625 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:10:05.105870 18098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.101458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.98.1:40313)
0924 23:10:05.101681 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.101715 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.102249 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.102453 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.102456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.102464 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:05.102962 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.102966 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.103630 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.103631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.104397 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.104399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.104740 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.104741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.104845 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.105326 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:05.105326 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:05.105358 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:05.105380 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:05.105528 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:05.105531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:05.105533 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.105590 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.105667 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":114,"threads_started":1}
I20260924 23:10:05.106416 18096 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34109
I20260924 23:10:05.106518 18096 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:05.106704 18096 heartbeater.cc:511] Master 127.0.98.62:34109 requested a full tablet report, sending...
I20260924 23:10:05.107216 17908 ts_manager.cc:194] Registered new tserver with Master: fb56c06d864b46e4b89c9a3b9e79b6a7 (127.0.98.1:38105)
I20260924 23:10:05.107826 17908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40313
I20260924 23:10:05.108045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 17968
I20260924 23:10:05.108161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:10:05.108345 392 external_mini_cluster.cc:1428] Setting key bfde78ac4303fb8e0d3a20d8dfc7acd4
I20260924 23:10:05.109203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--builtin_ntp_servers=127.0.98.20:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:05.190333 18099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:05.190550 18099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:05.190575 18099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:05.191735 18099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:05.191777 18099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:05.191803 18099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:10:05.193138 18099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:05.193393 18099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:05.193696 18099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:05.198911 18105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:05.198938 18104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:05.198979 18099 server_base.cc:1034] running on GCE node
W20260924 23:10:05.198918 18107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:05.199325 18099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:05.199555 18099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:05.200711 18099 hybrid_clock.cc:648] HybridClock initialized: now 1790291405200689 us; error 39 us; skew 500 ppm
I20260924 23:10:05.202075 18099 webserver.cc:460] Webserver started at http://127.0.98.2:35795/ using document root <none> and password file <none>
I20260924 23:10:05.202287 18099 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:05.202329 18099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:05.202419 18099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:05.203431 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "c30fee4d9d67d443c7c4d681d67f48c6"
server_key_iv: "1731fd8b299884b406870ffc6622ccce"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.203735 18099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "c30fee4d9d67d443c7c4d681d67f48c6"
server_key_iv: "1731fd8b299884b406870ffc6622ccce"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.204964 18099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:10:05.205659 18113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.205834 18099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:05.205955 18099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "c30fee4d9d67d443c7c4d681d67f48c6"
server_key_iv: "1731fd8b299884b406870ffc6622ccce"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.206034 18099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:05.219888 18099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:05.220216 18099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:05.220330 18099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:05.220533 18099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:05.220912 18099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:05.220950 18099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.220997 18099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:05.221025 18099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.227404 18099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40145
I20260924 23:10:05.227499 18226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40145 every 8 connection(s)
I20260924 23:10:05.227840 18099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:10:05.233508 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.228688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50457 (local address 127.0.98.62:34109)
0924 23:10:05.228751 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.228753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.229044 (+ 291us) server_negotiation.cc:407] Connection header received
0924 23:10:05.229565 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.229567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.229578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.229647 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:05.230465 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.230957 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.232050 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.232241 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.232292 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.233281 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:05.233282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:05.233282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:05.233287 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:05.233298 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:05.233302 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:05.233344 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.233408 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.233432 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 23:10:05.233754 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.228655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.98.2:50457)
0924 23:10:05.228864 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.228910 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.229438 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.229721 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.229725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.229734 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:05.230375 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.230383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.231058 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.231060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.231969 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.231971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.232323 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.232324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.232446 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.233137 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:05.233137 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:05.233169 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:05.233197 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:05.233347 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:05.233349 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:05.233352 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.233405 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.233511 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260924 23:10:05.233850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18099
I20260924 23:10:05.233989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:10:05.234272 392 external_mini_cluster.cc:1428] Setting key e925c467b74dfe69edeefcabfc5562ec
I20260924 23:10:05.234400 18227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34109
I20260924 23:10:05.234530 18227 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:05.234738 18227 heartbeater.cc:511] Master 127.0.98.62:34109 requested a full tablet report, sending...
I20260924 23:10:05.235109 17908 ts_manager.cc:194] Registered new tserver with Master: 3e0eb63f38ab48bdb8dbb2c95e305dfc (127.0.98.2:40145)
I20260924 23:10:05.235186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--builtin_ntp_servers=127.0.98.20:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:10:05.235621 17908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50457
W20260924 23:10:05.315845 18230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:05.316084 18230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:05.316118 18230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:05.317306 18230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:05.317376 18230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:05.317422 18230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:10:05.318825 18230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:05.319104 18230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:05.319404 18230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:05.324704 18235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:05.324765 18236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:05.324708 18238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:05.324921 18230 server_base.cc:1034] running on GCE node
I20260924 23:10:05.325248 18230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:05.325459 18230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:05.326651 18230 hybrid_clock.cc:648] HybridClock initialized: now 1790291405326539 us; error 152 us; skew 500 ppm
I20260924 23:10:05.328279 18230 webserver.cc:460] Webserver started at http://127.0.98.3:44357/ using document root <none> and password file <none>
I20260924 23:10:05.328580 18230 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:05.328655 18230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:05.328843 18230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:05.330118 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c1f0577bcb54a6996451d6cea60cd3a"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "954fa1d6d2084415a67d60bcfc6e5d1c"
server_key_iv: "4ef2485ff53ec51caa535c1977c2f999"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.330476 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c1f0577bcb54a6996451d6cea60cd3a"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "954fa1d6d2084415a67d60bcfc6e5d1c"
server_key_iv: "4ef2485ff53ec51caa535c1977c2f999"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.331815 18230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:05.332615 18244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.332787 18230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:05.332890 18230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "9c1f0577bcb54a6996451d6cea60cd3a"
format_stamp: "Formatted at 2026-09-24 23:10:05 on dist-test-slave-ph9j"
server_key: "954fa1d6d2084415a67d60bcfc6e5d1c"
server_key_iv: "4ef2485ff53ec51caa535c1977c2f999"
server_key_version: "encryptionkey@0"
I20260924 23:10:05.332975 18230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:05.343204 18230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:05.343509 18230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:05.343659 18230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:05.343874 18230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:05.344223 18230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:05.344281 18230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.344323 18230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:05.344352 18230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:05.349987 18230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43225
I20260924 23:10:05.350054 18357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43225 every 8 connection(s)
I20260924 23:10:05.350394 18230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:10:05.355552 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.351225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60377 (local address 127.0.98.62:34109)
0924 23:10:05.351295 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.351297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.351514 (+ 217us) server_negotiation.cc:407] Connection header received
0924 23:10:05.351980 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.351981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.351991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.352051 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:05.352695 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.353181 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.354248 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.354461 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.354528 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.355300 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:05.355301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:05.355301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:05.355305 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:05.355319 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:05.355323 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:05.355368 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.355464 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.355491 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:10:05.355741 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.351181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34109 (local address 127.0.98.3:60377)
0924 23:10:05.351381 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.351422 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.351901 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.352123 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.352126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.352134 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:05.352626 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.352630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.353269 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.353269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.354141 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.354143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.354545 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.354546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.354650 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.355168 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:05.355168 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:05.355196 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:05.355218 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:05.355380 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:05.355385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:05.355388 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.355439 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.355516 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260924 23:10:05.356299 18358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34109
I20260924 23:10:05.356406 18358 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:05.356581 18358 heartbeater.cc:511] Master 127.0.98.62:34109 requested a full tablet report, sending...
I20260924 23:10:05.357028 17908 ts_manager.cc:194] Registered new tserver with Master: 9c1f0577bcb54a6996451d6cea60cd3a (127.0.98.3:43225)
I20260924 23:10:05.357396 17908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:60377
I20260924 23:10:05.360044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18230
I20260924 23:10:05.360123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:10:05.360311 392 external_mini_cluster.cc:1428] Setting key bf658bfcf8226e3f8c574a96d6447736
I20260924 23:10:05.361231 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:10:05.365227 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.362680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.0.98.62:34109)
0924 23:10:05.362725 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.362727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.362846 (+ 119us) server_negotiation.cc:407] Connection header received
0924 23:10:05.362893 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.362893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.362903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.362948 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:05.363103 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.363541 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.364247 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.364415 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.364463 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.365003 (+ 540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:05.365004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:05.365004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:05.365008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:05.365019 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:05.365023 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:05.365063 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.365109 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.365130 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:05.366745 17908 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:10:05.368180 17908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:10:05.374393 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.370268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43225 (local address 127.0.0.1:57100)
0924 23:10:05.370522 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.370534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.370582 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.371174 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.371174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.371176 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:05.371222 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.371224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.372032 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.372032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.373294 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.373297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.374076 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.374084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.374165 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.374179 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.374249 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.374300 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
I20260924 23:10:05.374487 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.370691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.0.98.1:38105)
0924 23:10:05.370848 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.370853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.370863 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:05.370902 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.370904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.370918 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.370998 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:05.371173 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.372239 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.373328 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.374172 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.374246 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.374331 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.374360 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.374396 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 23:10:05.374692 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.370343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.0.98.2:40145)
0924 23:10:05.370723 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.370728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.370738 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:05.370775 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.370777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.370791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.370907 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:05.371076 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.372216 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.373457 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.374170 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.374457 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.374549 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.374576 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.374615 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":83,"threads_started":1}
I20260924 23:10:05.374519 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.370241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40145 (local address 127.0.0.1:41282)
0924 23:10:05.370445 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.370486 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.370550 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.370918 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.370922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.370923 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:05.370991 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.370993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.372312 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.372313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.373382 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.373384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.374264 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.374265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.374326 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.374331 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.374362 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.374412 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260924 23:10:05.374605 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.370480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38105 (local address 127.0.0.1:53048)
0924 23:10:05.370642 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.370654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.370697 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.371066 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.371066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.371068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:05.371097 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.371098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.372323 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.372324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.373243 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.373245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.374252 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.374253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.374304 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.374307 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.374348 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.374383 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260924 23:10:05.375077 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.370348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.0.98.3:43225)
0924 23:10:05.371022 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.371026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.371035 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:05.371069 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.371071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.371095 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.371165 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:05.371275 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.371944 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.373392 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.373992 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.374074 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.374168 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.374922 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.374965 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":83,"threads_started":1}
I20260924 23:10:05.375687 18161 tablet_service.cc:1467] Processing CreateTablet for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 (DEFAULT_TABLE table=test-table [id=bdc0f84452c24306beec2098080e0fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:05.375689 18030 tablet_service.cc:1467] Processing CreateTablet for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 (DEFAULT_TABLE table=test-table [id=bdc0f84452c24306beec2098080e0fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:05.376044 18030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:05.376044 18161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:05.379554 18372 tablet_bootstrap.cc:492] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Bootstrap starting.
I20260924 23:10:05.379948 18292 tablet_service.cc:1467] Processing CreateTablet for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 (DEFAULT_TABLE table=test-table [id=bdc0f84452c24306beec2098080e0fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:05.380216 18372 tablet_bootstrap.cc:654] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:05.380270 18292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:05.380515 18372 log.cc:824] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:05.380910 18373 tablet_bootstrap.cc:492] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: Bootstrap starting.
I20260924 23:10:05.381333 18372 tablet_bootstrap.cc:492] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: No bootstrap required, opened a new log
I20260924 23:10:05.381397 18372 ts_tablet_manager.cc:1397] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:05.381597 18373 tablet_bootstrap.cc:654] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:05.381997 18373 log.cc:824] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:05.382833 18373 tablet_bootstrap.cc:492] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: No bootstrap required, opened a new log
I20260924 23:10:05.382905 18373 ts_tablet_manager.cc:1397] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:05.383275 18376 tablet_bootstrap.cc:492] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Bootstrap starting.
I20260924 23:10:05.383912 18376 tablet_bootstrap.cc:654] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:05.384212 18376 log.cc:824] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:05.384297 18372 raft_consensus.cc:348] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.384481 18372 raft_consensus.cc:374] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:05.384565 18372 raft_consensus.cc:729] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e0eb63f38ab48bdb8dbb2c95e305dfc, State: Initialized, Role: FOLLOWER
I20260924 23:10:05.384717 18372 consensus_queue.cc:260] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.384933 18376 tablet_bootstrap.cc:492] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: No bootstrap required, opened a new log
I20260924 23:10:05.384979 18372 ts_tablet_manager.cc:1428] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 23:10:05.385003 18376 ts_tablet_manager.cc:1397] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:05.385114 18227 heartbeater.cc:503] Master 127.0.98.62:34109 was elected leader, sending a full tablet report...
I20260924 23:10:05.385460 18373 raft_consensus.cc:348] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.385617 18373 raft_consensus.cc:374] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:05.385659 18373 raft_consensus.cc:729] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb56c06d864b46e4b89c9a3b9e79b6a7, State: Initialized, Role: FOLLOWER
I20260924 23:10:05.385761 18373 consensus_queue.cc:260] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.386060 18373 ts_tablet_manager.cc:1428] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 23:10:05.386219 18096 heartbeater.cc:503] Master 127.0.98.62:34109 was elected leader, sending a full tablet report...
I20260924 23:10:05.387504 18376 raft_consensus.cc:348] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.387621 18376 raft_consensus.cc:374] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:05.387642 18376 raft_consensus.cc:729] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1f0577bcb54a6996451d6cea60cd3a, State: Initialized, Role: FOLLOWER
I20260924 23:10:05.387768 18376 consensus_queue.cc:260] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.388033 18376 ts_tablet_manager.cc:1428] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:10:05.388056 18358 heartbeater.cc:503] Master 127.0.98.62:34109 was elected leader, sending a full tablet report...
W20260924 23:10:05.478448 18228 tablet.cc:2369] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:10:05.600997 18359 tablet.cc:2369] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:10:05.601397 18097 tablet.cc:2369] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:05.630810 18379 raft_consensus.cc:482] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:10:05.631000 18379 raft_consensus.cc:504] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.631399 18379 leader_election.cc:288] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e0eb63f38ab48bdb8dbb2c95e305dfc (127.0.98.2:40145), 9c1f0577bcb54a6996451d6cea60cd3a (127.0.98.3:43225)
I20260924 23:10:05.634533 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.631700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51447 (local address 127.0.98.3:43225)
0924 23:10:05.631753 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.631755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.631865 (+ 110us) server_negotiation.cc:407] Connection header received
0924 23:10:05.631987 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.631988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.631996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.632061 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:05.632222 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.632688 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.633706 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.634165 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.634245 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.634255 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.634406 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.634455 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:10:05.634534 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.631723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41997 (local address 127.0.98.2:40145)
0924 23:10:05.631776 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.631778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.631873 (+ 95us) server_negotiation.cc:407] Connection header received
0924 23:10:05.632008 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.632009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.632019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.632099 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:05.632277 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.632734 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.633716 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.634165 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.634245 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.634256 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.634409 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.634455 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:10:05.634560 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.631659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40145 (local address 127.0.98.1:41997)
0924 23:10:05.631783 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.631793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.631885 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.632102 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.632103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.632104 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:05.632186 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.632188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.632822 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.632823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.633618 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.633620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.634259 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.634260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.634335 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.634338 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.634403 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.634465 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260924 23:10:05.634572 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.631640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43225 (local address 127.0.98.1:51447)
0924 23:10:05.631776 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.631788 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:05.631877 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:05.632066 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:05.632067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:05.632069 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:05.632143 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.632145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.632776 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.632777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.633608 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:05.633610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.634257 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.634258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:05.634333 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.634337 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:05.634401 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:10:05.634478 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":76,"thread_start_us":68,"threads_started":1}
I20260924 23:10:05.634908 18181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b65d64e8c5c4dd1bfe8e805fc4c3dc4" candidate_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" is_pre_election: true
I20260924 23:10:05.635092 18181 raft_consensus.cc:2454] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb56c06d864b46e4b89c9a3b9e79b6a7 in term 0.
I20260924 23:10:05.635104 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b65d64e8c5c4dd1bfe8e805fc4c3dc4" candidate_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" is_pre_election: true
I20260924 23:10:05.635275 18312 raft_consensus.cc:2454] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb56c06d864b46e4b89c9a3b9e79b6a7 in term 0.
I20260924 23:10:05.635368 17985 leader_election.cc:302] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e0eb63f38ab48bdb8dbb2c95e305dfc, fb56c06d864b46e4b89c9a3b9e79b6a7; no voters:
I20260924 23:10:05.635506 18379 raft_consensus.cc:2781] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:10:05.635565 18379 raft_consensus.cc:482] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:10:05.635589 18379 raft_consensus.cc:3037] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:05.636543 18379 raft_consensus.cc:504] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.636758 18379 leader_election.cc:288] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [CANDIDATE]: Term 1 election: Requested vote from peers 3e0eb63f38ab48bdb8dbb2c95e305dfc (127.0.98.2:40145), 9c1f0577bcb54a6996451d6cea60cd3a (127.0.98.3:43225)
I20260924 23:10:05.636895 18181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b65d64e8c5c4dd1bfe8e805fc4c3dc4" candidate_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc"
I20260924 23:10:05.636997 18181 raft_consensus.cc:3037] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:05.637033 18312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b65d64e8c5c4dd1bfe8e805fc4c3dc4" candidate_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1f0577bcb54a6996451d6cea60cd3a"
I20260924 23:10:05.637120 18312 raft_consensus.cc:3037] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:05.637785 18181 raft_consensus.cc:2454] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb56c06d864b46e4b89c9a3b9e79b6a7 in term 1.
I20260924 23:10:05.637848 18312 raft_consensus.cc:2454] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb56c06d864b46e4b89c9a3b9e79b6a7 in term 1.
I20260924 23:10:05.638026 17983 leader_election.cc:302] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c1f0577bcb54a6996451d6cea60cd3a, fb56c06d864b46e4b89c9a3b9e79b6a7; no voters:
I20260924 23:10:05.638137 18379 raft_consensus.cc:2781] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:05.638281 18379 raft_consensus.cc:686] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 1 LEADER]: Becoming Leader. State: Replica: fb56c06d864b46e4b89c9a3b9e79b6a7, State: Running, Role: LEADER
I20260924 23:10:05.638387 18379 consensus_queue.cc:237] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } }
I20260924 23:10:05.639405 17907 catalog_manager.cc:5495] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb56c06d864b46e4b89c9a3b9e79b6a7 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb56c06d864b46e4b89c9a3b9e79b6a7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38105 } health_report { overall_health: HEALTHY } } }
I20260924 23:10:05.657509 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.654951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53052 (local address 127.0.98.1:38105)
0924 23:10:05.655041 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:05.655043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:05.655053 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:05.655100 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:05.655101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:05.655124 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:05.655196 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:05.655369 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.655860 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.656698 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:05.656900 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:05.656964 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:05.657140 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:05.657247 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:05.657311 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:05.657384 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 23:10:05.657404 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 23:10:05.659655 18181 raft_consensus.cc:1261] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 1 FOLLOWER]: Refusing update from remote peer fb56c06d864b46e4b89c9a3b9e79b6a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:05.659669 18312 raft_consensus.cc:1261] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 1 FOLLOWER]: Refusing update from remote peer fb56c06d864b46e4b89c9a3b9e79b6a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:05.660087 18383 consensus_queue.cc:1035] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1f0577bcb54a6996451d6cea60cd3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:05.660240 18379 consensus_queue.cc:1035] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e0eb63f38ab48bdb8dbb2c95e305dfc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:05.663228 18386 mvcc.cc:205] Tried to move back new op lower bound from 7333033597578919936 to 7333033597495717888. Current Snapshot: MvccSnapshot[applied={T|T < 7333033597578919936}]
I20260924 23:10:05.663501 18388 mvcc.cc:205] Tried to move back new op lower bound from 7333033597578919936 to 7333033597495717888. Current Snapshot: MvccSnapshot[applied={T|T < 7333033597578919936}]
I20260924 23:10:05.664407 18387 mvcc.cc:205] Tried to move back new op lower bound from 7333033597578919936 to 7333033597495717888. Current Snapshot: MvccSnapshot[applied={T|T < 7333033597578919936 or (T in {7333033597578919936})}]
I20260924 23:10:05.665822 17907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:05.665966 17907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:05.666839 17907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26a5f11a71ac45a6878c1b839b201115: Sending DeleteTablet for 3 replicas of tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4
I20260924 23:10:05.667258 18030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:05 UTC) from {username='slave'} at 127.0.0.1:53048
I20260924 23:10:05.667251 18161 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:05 UTC) from {username='slave'} at 127.0.0.1:41282
I20260924 23:10:05.667496 18396 tablet_replica.cc:330] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: stopping tablet replica
I20260924 23:10:05.667485 18292 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:05 UTC) from {username='slave'} at 127.0.0.1:57100
I20260924 23:10:05.667601 18396 raft_consensus.cc:2229] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:05.667719 18396 raft_consensus.cc:2258] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:05.667735 18398 tablet_replica.cc:330] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: stopping tablet replica
I20260924 23:10:05.667793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17968
I20260924 23:10:05.667886 18397 tablet_replica.cc:330] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7: stopping tablet replica
I20260924 23:10:05.667977 18397 raft_consensus.cc:2229] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:10:05.668079 18397 raft_consensus.cc:2258] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P fb56c06d864b46e4b89c9a3b9e79b6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:05.669121 18398 raft_consensus.cc:2229] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:05.669258 18398 raft_consensus.cc:2258] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:05.669701 18398 ts_tablet_manager.cc:1905] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:05.670612 18396 ts_tablet_manager.cc:1905] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:05.671885 18398 ts_tablet_manager.cc:1918] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:05.671980 18398 log.cc:1196] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/2b65d64e8c5c4dd1bfe8e805fc4c3dc4
I20260924 23:10:05.672267 18398 ts_tablet_manager.cc:1939] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 9c1f0577bcb54a6996451d6cea60cd3a: Deleting consensus metadata
I20260924 23:10:05.672971 18396 ts_tablet_manager.cc:1918] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:05.673071 18396 log.cc:1196] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/2b65d64e8c5c4dd1bfe8e805fc4c3dc4
I20260924 23:10:05.673094 17892 catalog_manager.cc:4841] TS 9c1f0577bcb54a6996451d6cea60cd3a (127.0.98.3:43225): tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 (table test-table [id=bdc0f84452c24306beec2098080e0fc7]) successfully deleted
I20260924 23:10:05.673347 18396 ts_tablet_manager.cc:1939] T 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 P 3e0eb63f38ab48bdb8dbb2c95e305dfc: Deleting consensus metadata
I20260924 23:10:05.673826 17894 catalog_manager.cc:4841] TS 3e0eb63f38ab48bdb8dbb2c95e305dfc (127.0.98.2:40145): tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4 (table test-table [id=bdc0f84452c24306beec2098080e0fc7]) successfully deleted
I20260924 23:10:05.675910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18099
W20260924 23:10:05.676663 17894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38105 (error 108)
I20260924 23:10:05.677325 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:05.677155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38105 (local address 127.0.0.1:53058)
0924 23:10:05.677226 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:05.677280 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 23:10:05.677485 17894 catalog_manager.cc:4584] TS fb56c06d864b46e4b89c9a3b9e79b6a7 (127.0.98.1:38105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b65d64e8c5c4dd1bfe8e805fc4c3dc4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38105: connect: Connection refused (error 111)
I20260924 23:10:05.681545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18230
I20260924 23:10:05.687034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 17877
2026-09-24T23:10:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (890 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:05 dist-test-slave-ph9j krb5kdc[18402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:05 dist-test-slave-ph9j krb5kdc[18402](info): set up 2 sockets
Sep 24 23:10:05 dist-test-slave-ph9j krb5kdc[18402](info): commencing operation
krb5kdc: starting...
W20260924 23:10:06.749605 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:06 dist-test-slave-ph9j krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:06.869545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:06.952575 18415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:06.952786 18415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:06.952809 18415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:06.954025 18415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:06.954066 18415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:06.954080 18415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:06.954093 18415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:06.954106 18415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:06.955549 18415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:06.955739 18415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:06.955904 18415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:06.960759 18423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:06.960769 18421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:06.960870 18415 server_base.cc:1034] running on GCE node
W20260924 23:10:06.960777 18420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:06.961244 18415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:06.961561 18415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:06.962721 18415 hybrid_clock.cc:648] HybridClock initialized: now 1790291406962707 us; error 33 us; skew 500 ppm
Sep 24 23:10:06 dist-test-slave-ph9j krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:06.965301 18415 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:06.965808 18415 webserver.cc:460] Webserver started at http://127.0.98.62:36241/ using document root <none> and password file <none>
I20260924 23:10:06.966079 18415 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:06.966128 18415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:06.966274 18415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:06.967361 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "31e88fcd3efb417e95aceebeb1c2a8ab"
format_stamp: "Formatted at 2026-09-24 23:10:06 on dist-test-slave-ph9j"
server_key: "bbd2d7f6bb87dbe3a74a3a4dfaced368"
server_key_iv: "8666bfbbc56fc7544e277f97302b1d34"
server_key_version: "encryptionkey@0"
I20260924 23:10:06.967744 18415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "31e88fcd3efb417e95aceebeb1c2a8ab"
format_stamp: "Formatted at 2026-09-24 23:10:06 on dist-test-slave-ph9j"
server_key: "bbd2d7f6bb87dbe3a74a3a4dfaced368"
server_key_iv: "8666bfbbc56fc7544e277f97302b1d34"
server_key_version: "encryptionkey@0"
I20260924 23:10:06.969105 18415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:06.969869 18430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:06.970082 18415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:06.970175 18415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "31e88fcd3efb417e95aceebeb1c2a8ab"
format_stamp: "Formatted at 2026-09-24 23:10:06 on dist-test-slave-ph9j"
server_key: "bbd2d7f6bb87dbe3a74a3a4dfaced368"
server_key_iv: "8666bfbbc56fc7544e277f97302b1d34"
server_key_version: "encryptionkey@0"
I20260924 23:10:06.970252 18415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:06.998679 18415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:07.000829 18415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:07.000994 18415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:07.005378 18415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34163
I20260924 23:10:07.005439 18482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34163 every 8 connection(s)
I20260924 23:10:07.005798 18415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:07.006831 18483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:07.009239 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: Bootstrap starting.
I20260924 23:10:07.009902 18483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:07.010212 18483 log.cc:824] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:07.010946 18483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: No bootstrap required, opened a new log
I20260924 23:10:07.012568 18483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260924 23:10:07.012709 18483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:07.012732 18483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31e88fcd3efb417e95aceebeb1c2a8ab, State: Initialized, Role: FOLLOWER
I20260924 23:10:07.012916 18483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260924 23:10:07.012982 18483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:07.013025 18483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:07.013077 18483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:07.013620 18483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260924 23:10:07.013736 18483 leader_election.cc:302] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31e88fcd3efb417e95aceebeb1c2a8ab; no voters:
I20260924 23:10:07.013975 18483 leader_election.cc:288] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:07.014015 18486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:07.014175 18486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [term 1 LEADER]: Becoming Leader. State: Replica: 31e88fcd3efb417e95aceebeb1c2a8ab, State: Running, Role: LEADER
I20260924 23:10:07.014262 18483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:07.014299 18486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260924 23:10:07.014835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18415
I20260924 23:10:07.014900 18488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31e88fcd3efb417e95aceebeb1c2a8ab. Latest consensus state: current_term: 1 leader_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260924 23:10:07.014959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:07.015002 18488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:07.015177 392 external_mini_cluster.cc:1428] Setting key 91f8fddc91adf1c98d601067d0e4f942
I20260924 23:10:07.015367 18487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31e88fcd3efb417e95aceebeb1c2a8ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260924 23:10:07.015467 18487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab [sys.catalog]: This master's current role is: LEADER
W20260924 23:10:07.016111 18502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:07.016168 18502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:07.016250 18495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:07.016397 18495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:07.017992 18495 catalog_manager.cc:1347] Generated new cluster ID: 48e6f242ce8e458084b75155876cf294
I20260924 23:10:07.018064 18495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:07.027462 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.015895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.0.98.62:34163)
0924 23:10:07.016088 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.016098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.016225 (+ 127us) server_negotiation.cc:407] Connection header received
0924 23:10:07.016387 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.016395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.016486 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.016640 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:07.017393 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.018819 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.019901 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.020539 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.020658 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.023633 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:07.023638 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:07.023638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:07.023651 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:07.025698 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.026165 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.026168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.026168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.026271 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.026490 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.026491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.026491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.026808 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:07.026947 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.027088 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.027226 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260924 23:10:07.035359 18495 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:07.035902 18495 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:07.050676 18495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: Generated new TSK 0
I20260924 23:10:07.050913 18495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:10:07.085201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:07.168248 18509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:07.168497 18509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:07.168538 18509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:07.169745 18509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:07.169816 18509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:07.169860 18509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:10:07.171429 18509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:07.171697 18509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:07.171940 18509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:07.177115 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.177158 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.177135 18509 server_base.cc:1034] running on GCE node
W20260924 23:10:07.177232 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.177611 18509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:07.177860 18509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:07.179060 18509 hybrid_clock.cc:648] HybridClock initialized: now 1790291407179029 us; error 49 us; skew 500 ppm
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:07.181701 18509 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:10:07.182165 18509 webserver.cc:460] Webserver started at http://127.0.98.1:44963/ using document root <none> and password file <none>
I20260924 23:10:07.182348 18509 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:07.182390 18509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:07.182479 18509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:07.183459 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "111ca0d5ef35429fa52260ba5dd3a670"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "d061cca2d4978b7dc21399e561179983"
server_key_iv: "c6a8cb379bc599f9ff2aead6e088d7b6"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.183758 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "111ca0d5ef35429fa52260ba5dd3a670"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "d061cca2d4978b7dc21399e561179983"
server_key_iv: "c6a8cb379bc599f9ff2aead6e088d7b6"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.185032 18509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:10:07.185737 18524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.185935 18509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:07.186015 18509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "111ca0d5ef35429fa52260ba5dd3a670"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "d061cca2d4978b7dc21399e561179983"
server_key_iv: "c6a8cb379bc599f9ff2aead6e088d7b6"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.186070 18509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:07.194953 18509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:07.196985 18509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:07.197106 18509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:07.197300 18509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:07.197764 18509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:07.197804 18509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.197854 18509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:07.197975 18509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.204370 18509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44023
I20260924 23:10:07.204445 18637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44023 every 8 connection(s)
I20260924 23:10:07.204800 18509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:10:07.210673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18509
I20260924 23:10:07.210821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:10:07.211038 392 external_mini_cluster.cc:1428] Setting key fa4be688febda157e839b3cf4b3db3a9
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:07.214919 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.205567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36183 (local address 127.0.98.62:34163)
0924 23:10:07.205660 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.205663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.205851 (+ 188us) server_negotiation.cc:407] Connection header received
0924 23:10:07.206807 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.206808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.206822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.206885 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:07.207501 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.208012 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.209010 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.209209 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.209262 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.211629 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:07.211630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:07.211631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:07.211637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:07.213332 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.213726 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.213727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.213727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.213790 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.214220 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.214221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.214221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.214353 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:07.214439 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.214729 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.214831 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 23:10:07.215107 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.205524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.1:36183)
0924 23:10:07.205721 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.205756 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.206683 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.206941 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.206945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.207206 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:07.207398 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.207402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.208100 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.208100 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.208931 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.208933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.209291 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.209292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.209400 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.209956 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:07.209966 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:07.211536 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:07.213421 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:07.213428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:07.213429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:07.213649 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:07.213859 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:07.213860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:07.213860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:07.214141 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:07.214440 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:07.214448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:07.214591 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.214694 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.214857 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":115,"threads_started":1}
I20260924 23:10:07.215778 18638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260924 23:10:07.215893 18638 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:07.216073 18638 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260924 23:10:07.216594 18447 ts_manager.cc:194] Registered new tserver with Master: 111ca0d5ef35429fa52260ba5dd3a670 (127.0.98.1:44023)
I20260924 23:10:07.217216 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36183
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:10:07.237517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:07.321455 18643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:07.321683 18643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:07.321712 18643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:07.323115 18643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:07.323161 18643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:07.323189 18643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:10:07.324731 18643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:07.324978 18643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:07.325224 18643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:07.330570 18649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.330653 18643 server_base.cc:1034] running on GCE node
W20260924 23:10:07.330574 18651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.330574 18648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.331024 18643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:07.331233 18643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:07.332369 18643 hybrid_clock.cc:648] HybridClock initialized: now 1790291407332360 us; error 33 us; skew 500 ppm
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:07.335279 18643 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:10:07.335726 18643 webserver.cc:460] Webserver started at http://127.0.98.2:38349/ using document root <none> and password file <none>
I20260924 23:10:07.335980 18643 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:07.336048 18643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:07.336205 18643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:07.337253 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7fc397d482a4f48b9de31a93ad11255"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "8e3c4ded3a3d65bcfc951beeb90c3834"
server_key_iv: "7ccf5163610a5ffa3934e75e70e1c749"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.337618 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7fc397d482a4f48b9de31a93ad11255"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "8e3c4ded3a3d65bcfc951beeb90c3834"
server_key_iv: "7ccf5163610a5ffa3934e75e70e1c749"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.339046 18643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:10:07.339823 18658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.340058 18643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:07.340171 18643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "b7fc397d482a4f48b9de31a93ad11255"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "8e3c4ded3a3d65bcfc951beeb90c3834"
server_key_iv: "7ccf5163610a5ffa3934e75e70e1c749"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.340263 18643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:07.356359 18643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:07.358410 18643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:07.358573 18643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:07.358812 18643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:07.359139 18643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:07.359194 18643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.359239 18643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:07.359261 18643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.366945 18643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35815
I20260924 23:10:07.366974 18771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35815 every 8 connection(s)
I20260924 23:10:07.367391 18643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:10:07.372882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18643
I20260924 23:10:07.373008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:10:07.373215 392 external_mini_cluster.cc:1428] Setting key a41667c710174f96d6bf31c49326121e
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:07.378273 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.368230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33513 (local address 127.0.98.62:34163)
0924 23:10:07.368310 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.368313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.368542 (+ 229us) server_negotiation.cc:407] Connection header received
0924 23:10:07.369584 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.369585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.369600 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.369650 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:07.370663 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.371383 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.372548 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.372819 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.372875 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.375529 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:07.375530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:07.375530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:07.375537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:07.376832 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.377241 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.377242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.377242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.377304 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.377556 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.377557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.377557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.377678 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:07.377755 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.378097 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.378180 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:10:07.378470 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.368231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.2:33513)
0924 23:10:07.368394 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.368457 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.369451 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.369785 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.369789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.370224 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:07.370539 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.370549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.371497 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.371498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.372467 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.372469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.372887 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.372888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.373058 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.373654 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:07.373665 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:07.375419 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:07.376921 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:07.376928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:07.376930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:07.377155 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:07.377374 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:07.377374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:07.377375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:07.377486 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:07.377745 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:07.377752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:07.377947 (+ 195us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.378077 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.378237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260924 23:10:07.378916 18772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260924 23:10:07.379014 18772 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:07.379209 18772 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260924 23:10:07.379602 18447 ts_manager.cc:194] Registered new tserver with Master: b7fc397d482a4f48b9de31a93ad11255 (127.0.98.2:35815)
I20260924 23:10:07.379988 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33513
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:10:07.400941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:07.481982 18777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:07.482183 18777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:07.482206 18777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:07.483292 18777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:07.483328 18777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:07.483353 18777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:10:07.484733 18777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:07.484915 18777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:07.485085 18777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:10:07.490109 18777 server_base.cc:1034] running on GCE node
W20260924 23:10:07.490057 18782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.490088 18785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.490057 18783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.490526 18777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:07.490743 18777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:07.491886 18777 hybrid_clock.cc:648] HybridClock initialized: now 1790291407491861 us; error 40 us; skew 500 ppm
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:07.494422 18777 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:10:07.494856 18777 webserver.cc:460] Webserver started at http://127.0.98.3:42261/ using document root <none> and password file <none>
I20260924 23:10:07.495077 18777 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:07.495146 18777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:07.495296 18777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:07.496340 18777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "0b8088f72eace27dced4a1923314a741"
server_key_iv: "201ad57f87c1a951ec863d0837b93da7"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.496703 18777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "0b8088f72eace27dced4a1923314a741"
server_key_iv: "201ad57f87c1a951ec863d0837b93da7"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.498040 18777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 23:10:07.498790 18792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.498965 18777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:07.499058 18777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "0b8088f72eace27dced4a1923314a741"
server_key_iv: "201ad57f87c1a951ec863d0837b93da7"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.499145 18777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:07.510089 18777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:07.511933 18777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:07.512092 18777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:07.512307 18777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:07.512607 18777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:07.512661 18777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.512701 18777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:07.512729 18777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.519106 18777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41743
I20260924 23:10:07.519153 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41743 every 8 connection(s)
I20260924 23:10:07.519443 18777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:10:07.525975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18777
I20260924 23:10:07.526260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:10:07.526475 392 external_mini_cluster.cc:1428] Setting key 21aaa2dd0486c857e4fe8bb8193e8d6b
Sep 24 23:10:07 dist-test-slave-ph9j krb5kdc[18402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:07.532583 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.520129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55553 (local address 127.0.98.62:34163)
0924 23:10:07.520194 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.520196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.520398 (+ 202us) server_negotiation.cc:407] Connection header received
0924 23:10:07.521824 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.521825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.521838 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.521966 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:07.522974 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.524081 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.525735 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.526301 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.526380 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.529286 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:07.529288 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:07.529288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:07.529296 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:07.531076 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.531540 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.531541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.531542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.531619 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.531928 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.531929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.531930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.532045 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:07.532103 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.532458 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.532521 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 23:10:07.532913 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.520102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.3:55553)
0924 23:10:07.520274 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.520316 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.521674 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.522116 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.522120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.522513 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:07.522788 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.522795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.524208 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.524209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.525636 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.525639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.526407 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.526408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.526581 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.527252 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:07.527263 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:07.529191 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:07.531152 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:07.531159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:07.531161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:07.531442 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:07.531712 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:07.531713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:07.531713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:07.531861 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:07.532120 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:07.532127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:07.532302 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.532437 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.532667 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260924 23:10:07.533548 18906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260924 23:10:07.533653 18906 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:07.533840 18906 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260924 23:10:07.534407 18447 ts_manager.cc:194] Registered new tserver with Master: a4e65ac1994e4ba1be4d6f2a36ec59c3 (127.0.98.3:41743)
I20260924 23:10:07.534914 18447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55553
I20260924 23:10:07.539773 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:10:07.549009 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.541950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.62:34163)
0924 23:10:07.541997 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.541999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.542008 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:07.542030 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.542031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.542040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.542093 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:07.542856 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.543403 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.544472 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.544654 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.544729 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.546524 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:07.546525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:07.546526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:07.546532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:07.547745 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.548145 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.548146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.548146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.548226 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:07.548446 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:07.548447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:07.548448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:07.548596 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:07.548710 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.548838 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.548903 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:10:07.550753 18447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:10:07.552420 18447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:10:07.558998 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.554819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41743 (local address 127.0.0.1:52726)
0924 23:10:07.555025 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.555069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.555135 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.555595 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.555599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.555602 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:07.555689 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.555691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.556748 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.556749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.557788 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.557791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.558699 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.558700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.558779 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.558794 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.558867 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.558920 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":57,"threads_started":1}
I20260924 23:10:07.559288 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.555198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.0.98.3:41743)
0924 23:10:07.555370 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.555375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.555387 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:10:07.555430 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.555432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.555448 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.555536 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:07.555827 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.556647 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.557860 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.558610 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.558689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.558746 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.559137 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.559190 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":107,"threads_started":1}
I20260924 23:10:07.559780 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.554849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35815 (local address 127.0.0.1:37780)
0924 23:10:07.555199 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.555208 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.555252 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.556022 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.556023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.556025 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:07.556067 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.556069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.557344 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.557345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.558310 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.558313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.559562 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.559563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.559624 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.559628 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.559675 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.559713 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":49,"thread_start_us":104,"threads_started":1}
I20260924 23:10:07.559945 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.555433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.0.98.1:44023)
0924 23:10:07.555690 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.555695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.555705 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:07.555742 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.555744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.555761 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.555836 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:07.556075 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.557191 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.558884 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.559574 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.559661 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.559717 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.559799 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.559839 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260924 23:10:07.560011 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.555035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.0.98.2:35815)
0924 23:10:07.555789 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.555794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.555804 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:07.555843 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.555845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.555861 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.555937 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:07.556145 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.557242 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.558400 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.559055 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.559272 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.559342 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.559885 (+ 543us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.559933 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":79,"threads_started":1}
I20260924 23:10:07.559989 18840 tablet_service.cc:1467] Processing CreateTablet for tablet 0b44d2ccf2de493d97885669df09f9df (DEFAULT_TABLE table=test-table [id=ba5fe20f7cf2410aaf82f65dbd514aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:07.560387 18840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b44d2ccf2de493d97885669df09f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:07.560701 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.555038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44023 (local address 127.0.0.1:35078)
0924 23:10:07.555609 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.555620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.555664 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.555955 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.555955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.555957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:07.555997 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.555999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.557337 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.557338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.558795 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.558798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.559661 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.559663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.559722 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.559728 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.559786 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.560632 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":32,"thread_start_us":90,"threads_started":1}
I20260924 23:10:07.560743 18705 tablet_service.cc:1467] Processing CreateTablet for tablet 0b44d2ccf2de493d97885669df09f9df (DEFAULT_TABLE table=test-table [id=ba5fe20f7cf2410aaf82f65dbd514aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:07.561093 18705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b44d2ccf2de493d97885669df09f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:07.561381 18572 tablet_service.cc:1467] Processing CreateTablet for tablet 0b44d2ccf2de493d97885669df09f9df (DEFAULT_TABLE table=test-table [id=ba5fe20f7cf2410aaf82f65dbd514aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:07.561679 18572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b44d2ccf2de493d97885669df09f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:07.563710 18923 tablet_bootstrap.cc:492] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Bootstrap starting.
I20260924 23:10:07.564253 18923 tablet_bootstrap.cc:654] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:07.564705 18925 tablet_bootstrap.cc:492] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: Bootstrap starting.
I20260924 23:10:07.564901 18923 log.cc:824] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:07.565254 18924 tablet_bootstrap.cc:492] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Bootstrap starting.
I20260924 23:10:07.565371 18925 tablet_bootstrap.cc:654] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:07.565656 18923 tablet_bootstrap.cc:492] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: No bootstrap required, opened a new log
I20260924 23:10:07.565738 18923 ts_tablet_manager.cc:1397] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:07.566004 18924 tablet_bootstrap.cc:654] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:07.566051 18925 log.cc:824] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:07.566684 18924 log.cc:824] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:07.566900 18925 tablet_bootstrap.cc:492] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: No bootstrap required, opened a new log
I20260924 23:10:07.566968 18925 ts_tablet_manager.cc:1397] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:07.567389 18924 tablet_bootstrap.cc:492] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: No bootstrap required, opened a new log
I20260924 23:10:07.567469 18924 ts_tablet_manager.cc:1397] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 23:10:07.567921 18923 raft_consensus.cc:348] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.568109 18923 raft_consensus.cc:374] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:07.568157 18923 raft_consensus.cc:729] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7fc397d482a4f48b9de31a93ad11255, State: Initialized, Role: FOLLOWER
I20260924 23:10:07.568290 18923 consensus_queue.cc:260] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.568573 18923 ts_tablet_manager.cc:1428] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:10:07.568710 18772 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260924 23:10:07.568984 18925 raft_consensus.cc:348] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.569160 18925 raft_consensus.cc:374] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:07.569201 18925 raft_consensus.cc:729] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111ca0d5ef35429fa52260ba5dd3a670, State: Initialized, Role: FOLLOWER
I20260924 23:10:07.569308 18925 consensus_queue.cc:260] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.569617 18925 ts_tablet_manager.cc:1428] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:10:07.569526 18924 raft_consensus.cc:348] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.569687 18924 raft_consensus.cc:374] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:07.569733 18924 raft_consensus.cc:729] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e65ac1994e4ba1be4d6f2a36ec59c3, State: Initialized, Role: FOLLOWER
I20260924 23:10:07.569772 18638 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260924 23:10:07.569839 18924 consensus_queue.cc:260] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.570133 18924 ts_tablet_manager.cc:1428] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 23:10:07.570210 18906 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
W20260924 23:10:07.618029 18773 tablet.cc:2369] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:07.650722 18931 raft_consensus.cc:482] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:10:07.650851 18931 raft_consensus.cc:504] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.651278 18931 leader_election.cc:288] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111ca0d5ef35429fa52260ba5dd3a670 (127.0.98.1:44023), b7fc397d482a4f48b9de31a93ad11255 (127.0.98.2:35815)
I20260924 23:10:07.655262 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.651723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51117 (local address 127.0.98.1:44023)
0924 23:10:07.651767 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.651769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.651921 (+ 152us) server_negotiation.cc:407] Connection header received
0924 23:10:07.651972 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.651974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.651987 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.652047 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:07.652218 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.652880 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.654271 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.654863 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.654945 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.654957 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.655157 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.655198 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 23:10:07.655262 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.651641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44419 (local address 127.0.98.2:35815)
0924 23:10:07.651690 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.651693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.651704 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:10:07.651738 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.651739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.651751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.651811 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:07.651990 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.652767 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.654271 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.654869 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.654945 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.654957 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.655144 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.655187 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:10:07.655262 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.651497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35815 (local address 127.0.98.3:44419)
0924 23:10:07.651565 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.651583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.651658 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.651821 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.651822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.651824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:07.651903 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.651905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.652845 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.652846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.654156 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.654159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.654962 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.654963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.655052 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.655059 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.655118 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.655161 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":4}
I20260924 23:10:07.655273 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.651638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44023 (local address 127.0.98.3:51117)
0924 23:10:07.651830 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.651839 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:07.651892 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:07.652071 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:07.652072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:07.652074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:07.652141 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.652142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.652953 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.652953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.654156 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:07.654159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.654962 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.654963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:07.655052 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.655059 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 23:10:07.655130 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 23:10:07.655161 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":11,"thread_start_us":134,"threads_started":1}
I20260924 23:10:07.655694 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b44d2ccf2de493d97885669df09f9df" candidate_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc397d482a4f48b9de31a93ad11255" is_pre_election: true
I20260924 23:10:07.655694 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b44d2ccf2de493d97885669df09f9df" candidate_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111ca0d5ef35429fa52260ba5dd3a670" is_pre_election: true
I20260924 23:10:07.655877 18592 raft_consensus.cc:2454] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4e65ac1994e4ba1be4d6f2a36ec59c3 in term 0.
I20260924 23:10:07.655886 18726 raft_consensus.cc:2454] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4e65ac1994e4ba1be4d6f2a36ec59c3 in term 0.
I20260924 23:10:07.656157 18795 leader_election.cc:302] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 111ca0d5ef35429fa52260ba5dd3a670, a4e65ac1994e4ba1be4d6f2a36ec59c3; no voters:
I20260924 23:10:07.656339 18931 raft_consensus.cc:2781] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:10:07.656405 18931 raft_consensus.cc:482] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:10:07.656440 18931 raft_consensus.cc:3037] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:07.657150 18931 raft_consensus.cc:504] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.657301 18931 leader_election.cc:288] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [CANDIDATE]: Term 1 election: Requested vote from peers 111ca0d5ef35429fa52260ba5dd3a670 (127.0.98.1:44023), b7fc397d482a4f48b9de31a93ad11255 (127.0.98.2:35815)
I20260924 23:10:07.657456 18726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b44d2ccf2de493d97885669df09f9df" candidate_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7fc397d482a4f48b9de31a93ad11255"
I20260924 23:10:07.657457 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b44d2ccf2de493d97885669df09f9df" candidate_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111ca0d5ef35429fa52260ba5dd3a670"
I20260924 23:10:07.657550 18592 raft_consensus.cc:3037] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:07.657557 18726 raft_consensus.cc:3037] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:07.658349 18726 raft_consensus.cc:2454] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4e65ac1994e4ba1be4d6f2a36ec59c3 in term 1.
I20260924 23:10:07.658416 18592 raft_consensus.cc:2454] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4e65ac1994e4ba1be4d6f2a36ec59c3 in term 1.
I20260924 23:10:07.658568 18796 leader_election.cc:302] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4e65ac1994e4ba1be4d6f2a36ec59c3, b7fc397d482a4f48b9de31a93ad11255; no voters:
I20260924 23:10:07.658692 18931 raft_consensus.cc:2781] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:07.658852 18931 raft_consensus.cc:686] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 1 LEADER]: Becoming Leader. State: Replica: a4e65ac1994e4ba1be4d6f2a36ec59c3, State: Running, Role: LEADER
I20260924 23:10:07.658963 18931 consensus_queue.cc:237] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } }
I20260924 23:10:07.659909 18446 catalog_manager.cc:5495] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4e65ac1994e4ba1be4d6f2a36ec59c3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e65ac1994e4ba1be4d6f2a36ec59c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 } health_report { overall_health: UNKNOWN } } }
I20260924 23:10:07.675611 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.673322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.0.98.3:41743)
0924 23:10:07.673387 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.673391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.673403 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:10:07.673437 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.673438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.673455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.673519 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:07.673647 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.674249 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.674959 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.675137 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.675184 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.675321 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:07.675406 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:07.675476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.675513 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.675520 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 23:10:07.677371 18592 raft_consensus.cc:1261] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 1 FOLLOWER]: Refusing update from remote peer a4e65ac1994e4ba1be4d6f2a36ec59c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:07.677378 18726 raft_consensus.cc:1261] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 1 FOLLOWER]: Refusing update from remote peer a4e65ac1994e4ba1be4d6f2a36ec59c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:07.677743 18931 consensus_queue.cc:1035] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111ca0d5ef35429fa52260ba5dd3a670" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:07.677932 18933 consensus_queue.cc:1035] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7fc397d482a4f48b9de31a93ad11255" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:07.680817 18937 mvcc.cc:205] Tried to move back new op lower bound from 7333033605843902464 to 7333033605771919360. Current Snapshot: MvccSnapshot[applied={T|T < 7333033605843902464}]
I20260924 23:10:07.681090 18936 mvcc.cc:205] Tried to move back new op lower bound from 7333033605843902464 to 7333033605771919360. Current Snapshot: MvccSnapshot[applied={T|T < 7333033605843902464}]
I20260924 23:10:07.681241 18938 mvcc.cc:205] Tried to move back new op lower bound from 7333033605843902464 to 7333033605771919360. Current Snapshot: MvccSnapshot[applied={T|T < 7333033605843902464 or (T in {7333033605843902464})}]
I20260924 23:10:07.684545 18447 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:07.684657 18447 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:07.685384 18447 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31e88fcd3efb417e95aceebeb1c2a8ab: Sending DeleteTablet for 3 replicas of tablet 0b44d2ccf2de493d97885669df09f9df
I20260924 23:10:07.685698 18840 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b44d2ccf2de493d97885669df09f9df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52726
I20260924 23:10:07.685742 18572 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b44d2ccf2de493d97885669df09f9df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35078
I20260924 23:10:07.685978 18705 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b44d2ccf2de493d97885669df09f9df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37780
I20260924 23:10:07.686060 18946 tablet_replica.cc:330] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670: stopping tablet replica
I20260924 23:10:07.686163 18946 raft_consensus.cc:2229] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:07.686271 18946 raft_consensus.cc:2258] T 0b44d2ccf2de493d97885669df09f9df P 111ca0d5ef35429fa52260ba5dd3a670 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:07.686317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18509
I20260924 23:10:07.686987 18947 tablet_replica.cc:330] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: stopping tablet replica
I20260924 23:10:07.687147 18947 raft_consensus.cc:2229] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:07.687266 18947 raft_consensus.cc:2258] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:07.687711 18947 ts_tablet_manager.cc:1905] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:07.689607 18947 ts_tablet_manager.cc:1918] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:07.689683 18947 log.cc:1196] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/0b44d2ccf2de493d97885669df09f9df
I20260924 23:10:07.685971 18945 tablet_replica.cc:330] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: stopping tablet replica
I20260924 23:10:07.689968 18947 ts_tablet_manager.cc:1939] T 0b44d2ccf2de493d97885669df09f9df P b7fc397d482a4f48b9de31a93ad11255: Deleting consensus metadata
I20260924 23:10:07.690101 18945 raft_consensus.cc:2229] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 23:10:07.690282 18945 raft_consensus.cc:2258] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:07.690368 18434 catalog_manager.cc:4841] TS b7fc397d482a4f48b9de31a93ad11255 (127.0.98.2:35815): tablet 0b44d2ccf2de493d97885669df09f9df (table test-table [id=ba5fe20f7cf2410aaf82f65dbd514aab]) successfully deleted
I20260924 23:10:07.690755 18945 ts_tablet_manager.cc:1905] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:07.692353 18945 ts_tablet_manager.cc:1918] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:07.692492 18945 log.cc:1196] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/0b44d2ccf2de493d97885669df09f9df
I20260924 23:10:07.692828 18945 ts_tablet_manager.cc:1939] T 0b44d2ccf2de493d97885669df09f9df P a4e65ac1994e4ba1be4d6f2a36ec59c3: Deleting consensus metadata
W20260924 23:10:07.692874 18433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44023 (error 108)
I20260924 23:10:07.693087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18643
I20260924 23:10:07.694849 18433 catalog_manager.cc:4841] TS a4e65ac1994e4ba1be4d6f2a36ec59c3 (127.0.98.3:41743): tablet 0b44d2ccf2de493d97885669df09f9df (table test-table [id=ba5fe20f7cf2410aaf82f65dbd514aab]) successfully deleted
I20260924 23:10:07.697358 18920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.693850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44023 (local address 127.0.0.1:35082)
0924 23:10:07.697271 (+ 3421us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:07.697314 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3398}
I20260924 23:10:07.697922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18777
W20260924 23:10:07.697919 18433 catalog_manager.cc:4584] TS 111ca0d5ef35429fa52260ba5dd3a670 (127.0.98.1:44023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b44d2ccf2de493d97885669df09f9df: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44023: connect: Connection refused (error 111)
I20260924 23:10:07.703011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18415
2026-09-24T23:10:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T23:10:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:07Z Disabled control of system clock
I20260924 23:10:07.729697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35983
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:07.810088 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:07.810329 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:07.810353 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:07.811484 18952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:07.811522 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:07.811534 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:07.811547 18952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:07.811559 18952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:07.812841 18952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:07.813055 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:07.813308 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:07.818380 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.818380 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.818439 18952 server_base.cc:1034] running on GCE node
W20260924 23:10:07.818593 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.818959 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:07.819217 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:07.820386 18952 hybrid_clock.cc:648] HybridClock initialized: now 1790291407820373 us; error 29 us; skew 500 ppm
I20260924 23:10:07.821615 18952 webserver.cc:460] Webserver started at http://127.0.98.62:33383/ using document root <none> and password file <none>
I20260924 23:10:07.821866 18952 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:07.821977 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:07.822160 18952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:07.823259 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "ae2557b8e47a431b956f92ce0261b76f"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "decc27462d88ef689937fe5d77bdda16"
server_key_iv: "7f1db0b345db9f42ab21a2808e7a56d3"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.823618 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae2557b8e47a431b956f92ce0261b76f"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "decc27462d88ef689937fe5d77bdda16"
server_key_iv: "7f1db0b345db9f42ab21a2808e7a56d3"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.824892 18952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:10:07.825587 18966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.825783 18952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:07.825870 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "ae2557b8e47a431b956f92ce0261b76f"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "decc27462d88ef689937fe5d77bdda16"
server_key_iv: "7f1db0b345db9f42ab21a2808e7a56d3"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.825964 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:07.835060 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:07.835356 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:07.835489 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:07.839398 18952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35983
I20260924 23:10:07.839457 19018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35983 every 8 connection(s)
I20260924 23:10:07.839775 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:07.840750 19019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:07.843111 19019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f: Bootstrap starting.
I20260924 23:10:07.843691 19019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:07.844000 19019 log.cc:824] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:07.843931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 18952
I20260924 23:10:07.844312 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:07.844578 392 external_mini_cluster.cc:1428] Setting key f4e60d6c07a2c542b31dd4775d97f03c
I20260924 23:10:07.845032 19019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f: No bootstrap required, opened a new log
I20260924 23:10:07.847592 19019 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2557b8e47a431b956f92ce0261b76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260924 23:10:07.847796 19019 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:07.847833 19019 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae2557b8e47a431b956f92ce0261b76f, State: Initialized, Role: FOLLOWER
I20260924 23:10:07.847985 19019 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2557b8e47a431b956f92ce0261b76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260924 23:10:07.848062 19019 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:07.848091 19019 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:07.848129 19019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:07.848794 19019 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2557b8e47a431b956f92ce0261b76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260924 23:10:07.848898 19019 leader_election.cc:302] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae2557b8e47a431b956f92ce0261b76f; no voters:
I20260924 23:10:07.849066 19019 leader_election.cc:288] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:07.849172 19024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:07.849331 19019 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:07.849339 19024 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [term 1 LEADER]: Becoming Leader. State: Replica: ae2557b8e47a431b956f92ce0261b76f, State: Running, Role: LEADER
I20260924 23:10:07.849452 19024 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2557b8e47a431b956f92ce0261b76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } }
I20260924 23:10:07.849776 19024 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae2557b8e47a431b956f92ce0261b76f. Latest consensus state: current_term: 1 leader_uuid: "ae2557b8e47a431b956f92ce0261b76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2557b8e47a431b956f92ce0261b76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } } }
I20260924 23:10:07.849855 19024 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:07.849789 19025 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae2557b8e47a431b956f92ce0261b76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae2557b8e47a431b956f92ce0261b76f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35983 } } }
I20260924 23:10:07.850011 19025 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:07.850273 19031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:07.851014 19031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:07.852778 19031 catalog_manager.cc:1347] Generated new cluster ID: 917443a78a6f4497baa2c5c1c729f631
I20260924 23:10:07.852828 19031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:07.852828 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:07.845193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.0.98.62:35983)
0924 23:10:07.845421 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:07.845433 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:07.845466 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:10:07.846595 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:07.846605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:07.846694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:07.846851 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:07.847114 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.848996 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.850185 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:07.850822 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:07.850925 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:07.852115 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:07.852119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:07.852120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:07.852332 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:07.852375 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:07.852390 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:07.852454 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:07.852538 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:10:07.852618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260924 23:10:07.873673 19031 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:07.874307 19031 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:07.893257 19031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f: Generated new TSK 0
I20260924 23:10:07.893486 19031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:07.900867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:07.981163 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:07.981362 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:07.981383 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:07.982465 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:07.982497 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:07.982522 19043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:10:07.983753 19043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:07.983933 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:07.984175 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:07.989277 19049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.989342 19051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:07.989318 19048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:07.989487 19043 server_base.cc:1034] running on GCE node
I20260924 23:10:07.989729 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:07.989984 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:07.991129 19043 hybrid_clock.cc:648] HybridClock initialized: now 1790291407991119 us; error 36 us; skew 500 ppm
I20260924 23:10:07.992316 19043 webserver.cc:460] Webserver started at http://127.0.98.1:44359/ using document root <none> and password file <none>
I20260924 23:10:07.992566 19043 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:07.992612 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:07.992775 19043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:07.993813 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "940d641c44094c52abf2d65d0cf06ae6"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "ad6e49a988cb750231ea76a6eece8173"
server_key_iv: "8b934998f146ff4ea9fc58686b11860b"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.994227 19043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "940d641c44094c52abf2d65d0cf06ae6"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "ad6e49a988cb750231ea76a6eece8173"
server_key_iv: "8b934998f146ff4ea9fc58686b11860b"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.995497 19043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:07.996179 19057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:07.996351 19043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:07.996452 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "940d641c44094c52abf2d65d0cf06ae6"
format_stamp: "Formatted at 2026-09-24 23:10:07 on dist-test-slave-ph9j"
server_key: "ad6e49a988cb750231ea76a6eece8173"
server_key_iv: "8b934998f146ff4ea9fc58686b11860b"
server_key_version: "encryptionkey@0"
I20260924 23:10:07.996526 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:08.023788 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:08.024140 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:08.024294 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:08.024540 19043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:08.024876 19043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:08.024933 19043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.024981 19043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:08.025013 19043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.030984 19043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46779
I20260924 23:10:08.031032 19170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46779 every 8 connection(s)
I20260924 23:10:08.031376 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:10:08.035895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19043
I20260924 23:10:08.036015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:10:08.036229 392 external_mini_cluster.cc:1428] Setting key 87446383a2e15f281bc05c8cc4e4ab59
I20260924 23:10:08.037106 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.032238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53311 (local address 127.0.98.62:35983)
0924 23:10:08.032305 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.032307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.032571 (+ 264us) server_negotiation.cc:407] Connection header received
0924 23:10:08.033038 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.033039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.033054 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.033123 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:08.033809 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.034425 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.035406 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.035592 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.035647 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.036785 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:08.036787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:08.036787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:08.036792 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:08.036809 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:08.036813 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:08.036880 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.036986 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.037012 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 23:10:08.037349 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.032201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35983 (local address 127.0.98.1:53311)
0924 23:10:08.032405 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.032446 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.032933 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.033206 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.033209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.033218 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:08.033706 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.033709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.034503 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.034504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.035311 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.035313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.035668 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.035669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.035852 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.036626 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:08.036626 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:08.036665 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:08.036699 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:08.036884 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:08.036891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:08.036894 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.036968 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.037103 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260924 23:10:08.037400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:10:08.037865 19171 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35983
I20260924 23:10:08.038035 19171 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:08.038237 19171 heartbeater.cc:511] Master 127.0.98.62:35983 requested a full tablet report, sending...
I20260924 23:10:08.038774 18983 ts_manager.cc:194] Registered new tserver with Master: 940d641c44094c52abf2d65d0cf06ae6 (127.0.98.1:46779)
I20260924 23:10:08.039407 18983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53311
W20260924 23:10:08.119478 19174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:08.119699 19174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:08.119721 19174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:08.120967 19174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:08.121016 19174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:08.121042 19174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:10:08.122511 19174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:08.122759 19174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:08.123061 19174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:08.128197 19180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:08.128232 19182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:08.128232 19179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:08.128400 19174 server_base.cc:1034] running on GCE node
I20260924 23:10:08.128610 19174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:08.128815 19174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:08.130009 19174 hybrid_clock.cc:648] HybridClock initialized: now 1790291408129991 us; error 34 us; skew 500 ppm
I20260924 23:10:08.131278 19174 webserver.cc:460] Webserver started at http://127.0.98.2:35613/ using document root <none> and password file <none>
I20260924 23:10:08.131546 19174 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:08.131633 19174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:08.131791 19174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:08.132884 19174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "080f7adf82c846b0b5b3148b15dc8a50"
format_stamp: "Formatted at 2026-09-24 23:10:08 on dist-test-slave-ph9j"
server_key: "ea0729be1c3ccf50bf457fcf063bfe30"
server_key_iv: "bcfd8f8a77b6988a8e574a1ed079440d"
server_key_version: "encryptionkey@0"
I20260924 23:10:08.133262 19174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "080f7adf82c846b0b5b3148b15dc8a50"
format_stamp: "Formatted at 2026-09-24 23:10:08 on dist-test-slave-ph9j"
server_key: "ea0729be1c3ccf50bf457fcf063bfe30"
server_key_iv: "bcfd8f8a77b6988a8e574a1ed079440d"
server_key_version: "encryptionkey@0"
I20260924 23:10:08.134691 19174 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:08.135403 19188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.135577 19174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:08.135696 19174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "080f7adf82c846b0b5b3148b15dc8a50"
format_stamp: "Formatted at 2026-09-24 23:10:08 on dist-test-slave-ph9j"
server_key: "ea0729be1c3ccf50bf457fcf063bfe30"
server_key_iv: "bcfd8f8a77b6988a8e574a1ed079440d"
server_key_version: "encryptionkey@0"
I20260924 23:10:08.135771 19174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:08.144376 19174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:08.144697 19174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:08.144855 19174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:08.145083 19174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:08.145409 19174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:08.145465 19174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.145500 19174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:08.145529 19174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.151834 19174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36105
I20260924 23:10:08.151872 19301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36105 every 8 connection(s)
I20260924 23:10:08.152218 19174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:10:08.153106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19174
I20260924 23:10:08.153227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:10:08.153465 392 external_mini_cluster.cc:1428] Setting key c02d03943616e57a956f55e52c11d41a
I20260924 23:10:08.154453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--builtin_ntp_servers=127.0.98.20:45169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:10:08.158733 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.152992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55287 (local address 127.0.98.62:35983)
0924 23:10:08.153075 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.153078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.153325 (+ 247us) server_negotiation.cc:407] Connection header received
0924 23:10:08.153980 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.153981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.153991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.154056 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:08.154869 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.155645 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.157188 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.157371 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.157425 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.158463 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:08.158464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:08.158464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:08.158469 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:08.158482 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:08.158486 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:08.158529 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.158652 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.158674 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 23:10:08.159051 19304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.152978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35983 (local address 127.0.98.2:55287)
0924 23:10:08.153179 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.153219 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.153851 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.154145 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.154150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.154161 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:08.154772 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.154777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.155754 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.155756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.157103 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.157105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.157451 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.157452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.157623 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.158293 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:08.158293 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:08.158336 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:08.158370 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:08.158557 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:08.158564 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:08.158568 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.158648 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.158773 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260924 23:10:08.159600 19302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35983
I20260924 23:10:08.159705 19302 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:08.159916 19302 heartbeater.cc:511] Master 127.0.98.62:35983 requested a full tablet report, sending...
I20260924 23:10:08.160310 18983 ts_manager.cc:194] Registered new tserver with Master: 080f7adf82c846b0b5b3148b15dc8a50 (127.0.98.2:36105)
I20260924 23:10:08.160677 18983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:55287
W20260924 23:10:08.236884 19305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:08.237104 19305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:08.237126 19305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:08.238389 19305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:08.238433 19305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:08.238459 19305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:10:08.239890 19305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:08.240144 19305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:08.240439 19305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:08.245357 19310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:08.245378 19313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:08.245332 19311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:08.245523 19305 server_base.cc:1034] running on GCE node
I20260924 23:10:08.245759 19305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:08.246019 19305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:08.247165 19305 hybrid_clock.cc:648] HybridClock initialized: now 1790291408247135 us; error 47 us; skew 500 ppm
I20260924 23:10:08.248353 19305 webserver.cc:460] Webserver started at http://127.0.98.3:44015/ using document root <none> and password file <none>
I20260924 23:10:08.248551 19305 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:08.248593 19305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:08.248683 19305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:08.249881 19305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "d325fb9e409846368412662f866016a2"
format_stamp: "Formatted at 2026-09-24 23:10:08 on dist-test-slave-ph9j"
server_key: "6a026a0ec270fcdb5c97d1eee9f3b843"
server_key_iv: "c2015cc04d6fa393508b187ea7a07b22"
server_key_version: "encryptionkey@0"
I20260924 23:10:08.250234 19305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d325fb9e409846368412662f866016a2"
format_stamp: "Formatted at 2026-09-24 23:10:08 on dist-test-slave-ph9j"
server_key: "6a026a0ec270fcdb5c97d1eee9f3b843"
server_key_iv: "c2015cc04d6fa393508b187ea7a07b22"
server_key_version: "encryptionkey@0"
I20260924 23:10:08.251506 19305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:08.252139 19319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.252292 19305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:08.252379 19305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "d325fb9e409846368412662f866016a2"
format_stamp: "Formatted at 2026-09-24 23:10:08 on dist-test-slave-ph9j"
server_key: "6a026a0ec270fcdb5c97d1eee9f3b843"
server_key_iv: "c2015cc04d6fa393508b187ea7a07b22"
server_key_version: "encryptionkey@0"
I20260924 23:10:08.252434 19305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:08.268360 19305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:08.268627 19305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:08.268728 19305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:08.268910 19305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:08.269186 19305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:08.269215 19305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.269237 19305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:08.269249 19305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:08.274777 19305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46657
I20260924 23:10:08.274883 19432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46657 every 8 connection(s)
I20260924 23:10:08.275207 19305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:10:08.279796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19305
I20260924 23:10:08.279883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:10:08.280036 392 external_mini_cluster.cc:1428] Setting key 40284024e85ad6f176bdfbc4c3d99269
I20260924 23:10:08.280658 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.275955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54307 (local address 127.0.98.62:35983)
0924 23:10:08.276034 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.276037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.276262 (+ 225us) server_negotiation.cc:407] Connection header received
0924 23:10:08.276721 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.276722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.276732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.276779 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:08.277394 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.277896 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.278936 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.279116 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.279165 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.280404 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:08.280406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:08.280406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:08.280412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:08.280428 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:08.280432 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:08.280480 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.280554 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.280581 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 23:10:08.280880 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.275907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35983 (local address 127.0.98.3:54307)
0924 23:10:08.276144 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.276179 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.276644 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.276840 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.276843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.276852 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:08.277330 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.277334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.278061 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.278062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.278863 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.278865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.279187 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.279188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.279293 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.280243 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:08.280244 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:08.280284 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:08.280318 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:08.280480 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:08.280485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:08.280488 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.280552 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.280652 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
I20260924 23:10:08.281261 19433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35983
I20260924 23:10:08.281376 19433 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:08.281565 19433 heartbeater.cc:511] Master 127.0.98.62:35983 requested a full tablet report, sending...
I20260924 23:10:08.281920 18983 ts_manager.cc:194] Registered new tserver with Master: d325fb9e409846368412662f866016a2 (127.0.98.3:46657)
I20260924 23:10:08.282284 18983 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54307
I20260924 23:10:08.291915 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:10:08.296420 19023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.293312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.0.98.62:35983)
0924 23:10:08.293388 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.293390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.293480 (+ 90us) server_negotiation.cc:407] Connection header received
0924 23:10:08.293562 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.293563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.293572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.293622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:08.293791 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.294367 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.295163 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.295368 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.295448 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.296108 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:08.296109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:08.296110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:08.296115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:08.296129 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:08.296133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:08.296245 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.296319 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.296330 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 23:10:08.298094 18983 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:10:08.299767 18983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:10:08.306540 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.302225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.0.1:41046)
0924 23:10:08.302396 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.302427 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.302525 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.303066 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.303069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.303071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:08.303144 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.303146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.304107 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.304108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.305194 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.305196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.306231 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.306232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.306313 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.306326 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.306389 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.306444 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":69,"threads_started":1}
I20260924 23:10:08.306571 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.302603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.0.98.2:36105)
0924 23:10:08.303208 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.303213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.303222 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:08.303265 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.303267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.303278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.303335 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:08.303448 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.304217 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.305489 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.306233 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.306304 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.306397 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.306429 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.306481 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":126,"threads_started":1}
I20260924 23:10:08.306913 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.302608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.98.1:46779)
0924 23:10:08.302873 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.302878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.302890 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:10:08.302931 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.302943 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.302960 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.303062 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:08.303214 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.304015 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.305280 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.306029 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.306102 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.306193 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.306777 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.306810 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":128,"threads_started":1}
I20260924 23:10:08.306545 19441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.302457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36105 (local address 127.0.0.1:39650)
0924 23:10:08.302832 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.302841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.302883 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.303358 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.303359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.303361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:08.303392 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.303393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.304312 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.304314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.305394 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.305396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.306312 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.306313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.306369 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.306374 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.306418 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.306457 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260924 23:10:08.307324 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.302225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46657 (local address 127.0.0.1:54642)
0924 23:10:08.302754 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.302764 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.302809 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.303102 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.303103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.303105 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:08.303152 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.303154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.305250 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.305251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.306071 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.306074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.307091 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.307092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.307154 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.307157 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.307214 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.307252 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":115,"threads_started":1}
I20260924 23:10:08.307612 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.302411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.98.3:46657)
0924 23:10:08.302932 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.302937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.302947 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:08.302991 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.302993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.303005 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.303104 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:08.304103 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.305162 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.306156 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.306903 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.306967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.307053 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.307480 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.307523 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":100,"threads_started":1}
I20260924 23:10:08.307636 19105 tablet_service.cc:1467] Processing CreateTablet for tablet e17b7e024dea47e88c985d53958b4006 (DEFAULT_TABLE table=test-table [id=968827f56c2a4eecb30e3bd252bd0d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:08.308033 19105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17b7e024dea47e88c985d53958b4006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:08.307971 19236 tablet_service.cc:1467] Processing CreateTablet for tablet e17b7e024dea47e88c985d53958b4006 (DEFAULT_TABLE table=test-table [id=968827f56c2a4eecb30e3bd252bd0d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:08.308141 19367 tablet_service.cc:1467] Processing CreateTablet for tablet e17b7e024dea47e88c985d53958b4006 (DEFAULT_TABLE table=test-table [id=968827f56c2a4eecb30e3bd252bd0d93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:08.308319 19236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17b7e024dea47e88c985d53958b4006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:08.308429 19367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17b7e024dea47e88c985d53958b4006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:08.310916 19447 tablet_bootstrap.cc:492] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6: Bootstrap starting.
I20260924 23:10:08.311745 19448 tablet_bootstrap.cc:492] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Bootstrap starting.
I20260924 23:10:08.311777 19447 tablet_bootstrap.cc:654] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:08.311930 19449 tablet_bootstrap.cc:492] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Bootstrap starting.
I20260924 23:10:08.312297 19447 log.cc:824] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:08.312501 19448 tablet_bootstrap.cc:654] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:08.312515 19449 tablet_bootstrap.cc:654] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:08.312831 19448 log.cc:824] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:08.312858 19449 log.cc:824] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:08.313258 19447 tablet_bootstrap.cc:492] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6: No bootstrap required, opened a new log
I20260924 23:10:08.313341 19447 ts_tablet_manager.cc:1397] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:08.313619 19448 tablet_bootstrap.cc:492] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: No bootstrap required, opened a new log
I20260924 23:10:08.313690 19448 ts_tablet_manager.cc:1397] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:08.314220 19449 tablet_bootstrap.cc:492] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: No bootstrap required, opened a new log
I20260924 23:10:08.314306 19449 ts_tablet_manager.cc:1397] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:08.316056 19449 raft_consensus.cc:348] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.316211 19449 raft_consensus.cc:374] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:08.316254 19449 raft_consensus.cc:729] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d325fb9e409846368412662f866016a2, State: Initialized, Role: FOLLOWER
I20260924 23:10:08.316358 19449 consensus_queue.cc:260] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.316356 19447 raft_consensus.cc:348] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.316532 19447 raft_consensus.cc:374] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:08.316577 19447 raft_consensus.cc:729] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940d641c44094c52abf2d65d0cf06ae6, State: Initialized, Role: FOLLOWER
I20260924 23:10:08.316495 19448 raft_consensus.cc:348] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.316638 19449 ts_tablet_manager.cc:1428] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:10:08.316649 19448 raft_consensus.cc:374] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:08.316692 19448 raft_consensus.cc:729] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080f7adf82c846b0b5b3148b15dc8a50, State: Initialized, Role: FOLLOWER
I20260924 23:10:08.316762 19433 heartbeater.cc:503] Master 127.0.98.62:35983 was elected leader, sending a full tablet report...
I20260924 23:10:08.316733 19447 consensus_queue.cc:260] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.316802 19448 consensus_queue.cc:260] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.317022 19447 ts_tablet_manager.cc:1428] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 23:10:08.317037 19448 ts_tablet_manager.cc:1428] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 23:10:08.317077 19171 heartbeater.cc:503] Master 127.0.98.62:35983 was elected leader, sending a full tablet report...
I20260924 23:10:08.317163 19302 heartbeater.cc:503] Master 127.0.98.62:35983 was elected leader, sending a full tablet report...
W20260924 23:10:08.402789 19303 tablet.cc:2369] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:08.442289 19454 raft_consensus.cc:482] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:10:08.442451 19454 raft_consensus.cc:504] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.442863 19454 leader_election.cc:288] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d325fb9e409846368412662f866016a2 (127.0.98.3:46657), 080f7adf82c846b0b5b3148b15dc8a50 (127.0.98.2:36105)
I20260924 23:10:08.445765 19173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.443068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36105 (local address 127.0.98.1:38657)
0924 23:10:08.443133 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.443147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.443191 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.443377 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.443378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.443380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:08.443459 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.443461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.444129 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.444130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.444977 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.444980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.445551 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.445553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.445623 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.445627 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.445680 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.445711 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":46}
I20260924 23:10:08.445955 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.443163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38657 (local address 127.0.98.2:36105)
0924 23:10:08.443234 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.443237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.443247 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:08.443283 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.443284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.443295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.443356 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:08.443532 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.444045 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.445068 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.445462 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.445517 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.445526 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.445798 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.445841 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:10:08.446054 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.443158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37109 (local address 127.0.98.3:46657)
0924 23:10:08.443214 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.443216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.443361 (+ 145us) server_negotiation.cc:407] Connection header received
0924 23:10:08.443389 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.443390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.443402 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.443461 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:08.443655 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.444130 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.445118 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.445514 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.445587 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.445596 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.445937 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.445982 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 23:10:08.446252 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.443070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46657 (local address 127.0.98.1:37109)
0924 23:10:08.443263 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.443273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:08.443337 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:08.443476 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:08.443477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:08.443479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:08.443567 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.443568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.444220 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.444221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.444992 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:08.444994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.445616 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.445617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:08.445814 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.445818 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:08.446140 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 23:10:08.446182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":13,"thread_start_us":94,"threads_started":1}
I20260924 23:10:08.446236 19256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17b7e024dea47e88c985d53958b4006" candidate_uuid: "940d641c44094c52abf2d65d0cf06ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080f7adf82c846b0b5b3148b15dc8a50" is_pre_election: true
I20260924 23:10:08.446404 19256 raft_consensus.cc:2454] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 940d641c44094c52abf2d65d0cf06ae6 in term 0.
I20260924 23:10:08.446596 19387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17b7e024dea47e88c985d53958b4006" candidate_uuid: "940d641c44094c52abf2d65d0cf06ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d325fb9e409846368412662f866016a2" is_pre_election: true
I20260924 23:10:08.446671 19059 leader_election.cc:302] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 080f7adf82c846b0b5b3148b15dc8a50, 940d641c44094c52abf2d65d0cf06ae6; no voters:
I20260924 23:10:08.446756 19387 raft_consensus.cc:2454] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 940d641c44094c52abf2d65d0cf06ae6 in term 0.
I20260924 23:10:08.446796 19454 raft_consensus.cc:2781] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:10:08.446846 19454 raft_consensus.cc:482] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:10:08.446872 19454 raft_consensus.cc:3037] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:08.447633 19454 raft_consensus.cc:504] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.447781 19454 leader_election.cc:288] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [CANDIDATE]: Term 1 election: Requested vote from peers d325fb9e409846368412662f866016a2 (127.0.98.3:46657), 080f7adf82c846b0b5b3148b15dc8a50 (127.0.98.2:36105)
I20260924 23:10:08.447988 19387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17b7e024dea47e88c985d53958b4006" candidate_uuid: "940d641c44094c52abf2d65d0cf06ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d325fb9e409846368412662f866016a2"
I20260924 23:10:08.448014 19256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17b7e024dea47e88c985d53958b4006" candidate_uuid: "940d641c44094c52abf2d65d0cf06ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "080f7adf82c846b0b5b3148b15dc8a50"
I20260924 23:10:08.448073 19387 raft_consensus.cc:3037] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:08.448081 19256 raft_consensus.cc:3037] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:08.449021 19256 raft_consensus.cc:2454] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 940d641c44094c52abf2d65d0cf06ae6 in term 1.
I20260924 23:10:08.449021 19387 raft_consensus.cc:2454] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 940d641c44094c52abf2d65d0cf06ae6 in term 1.
I20260924 23:10:08.449254 19058 leader_election.cc:302] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 940d641c44094c52abf2d65d0cf06ae6, d325fb9e409846368412662f866016a2; no voters:
I20260924 23:10:08.449339 19454 raft_consensus.cc:2781] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:08.449491 19454 raft_consensus.cc:686] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [term 1 LEADER]: Becoming Leader. State: Replica: 940d641c44094c52abf2d65d0cf06ae6, State: Running, Role: LEADER
I20260924 23:10:08.449596 19454 consensus_queue.cc:237] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } }
I20260924 23:10:08.450497 18981 catalog_manager.cc:5495] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 940d641c44094c52abf2d65d0cf06ae6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "940d641c44094c52abf2d65d0cf06ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "940d641c44094c52abf2d65d0cf06ae6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 } health_report { overall_health: UNKNOWN } } }
I20260924 23:10:08.487287 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.485027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.1:46779)
0924 23:10:08.485088 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:08.485090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:08.485116 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:10:08.485153 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:08.485154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:08.485173 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:08.485229 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:08.485338 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.485792 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.486561 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:08.486757 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:08.486815 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:08.486927 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:08.487015 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:08.487082 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:08.487131 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 23:10:08.487141 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:10:08.489220 19387 raft_consensus.cc:1261] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 1 FOLLOWER]: Refusing update from remote peer 940d641c44094c52abf2d65d0cf06ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:08.489221 19256 raft_consensus.cc:1261] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 1 FOLLOWER]: Refusing update from remote peer 940d641c44094c52abf2d65d0cf06ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:08.489634 19454 consensus_queue.cc:1035] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080f7adf82c846b0b5b3148b15dc8a50" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:08.489786 19457 consensus_queue.cc:1035] T e17b7e024dea47e88c985d53958b4006 P 940d641c44094c52abf2d65d0cf06ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d325fb9e409846368412662f866016a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:08.492700 19462 mvcc.cc:205] Tried to move back new op lower bound from 7333033609168670720 to 7333033609010282496. Current Snapshot: MvccSnapshot[applied={T|T < 7333033609168670720 or (T in {7333033609168670720})}]
I20260924 23:10:08.492892 19461 mvcc.cc:205] Tried to move back new op lower bound from 7333033609168670720 to 7333033609010282496. Current Snapshot: MvccSnapshot[applied={T|T < 7333033609168670720}]
I20260924 23:10:08.494946 18981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:08.495050 18981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:08.495754 18981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae2557b8e47a431b956f92ce0261b76f: Sending DeleteTablet for 3 replicas of tablet e17b7e024dea47e88c985d53958b4006
I20260924 23:10:08.496146 19236 tablet_service.cc:1514] Processing DeleteTablet for tablet e17b7e024dea47e88c985d53958b4006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:08 UTC) from {username='slave'} at 127.0.0.1:39650
I20260924 23:10:08.496160 19105 tablet_service.cc:1514] Processing DeleteTablet for tablet e17b7e024dea47e88c985d53958b4006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:08 UTC) from {username='slave'} at 127.0.0.1:41046
I20260924 23:10:08.496356 19471 tablet_replica.cc:330] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: stopping tablet replica
I20260924 23:10:08.496346 19367 tablet_service.cc:1514] Processing DeleteTablet for tablet e17b7e024dea47e88c985d53958b4006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:08 UTC) from {username='slave'} at 127.0.0.1:54642
I20260924 23:10:08.496447 19471 raft_consensus.cc:2229] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:08.496560 19471 raft_consensus.cc:2258] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:08.496726 19472 tablet_replica.cc:330] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: stopping tablet replica
I20260924 23:10:08.496809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19043
I20260924 23:10:08.496821 19472 raft_consensus.cc:2229] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:08.497973 19472 raft_consensus.cc:2258] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:08.498409 19472 ts_tablet_manager.cc:1905] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:08.499730 19471 ts_tablet_manager.cc:1905] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:08.500412 19472 ts_tablet_manager.cc:1918] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:08.500501 19472 log.cc:1196] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/e17b7e024dea47e88c985d53958b4006
I20260924 23:10:08.500734 19472 ts_tablet_manager.cc:1939] T e17b7e024dea47e88c985d53958b4006 P d325fb9e409846368412662f866016a2: Deleting consensus metadata
I20260924 23:10:08.501170 18967 catalog_manager.cc:4841] TS d325fb9e409846368412662f866016a2 (127.0.98.3:46657): tablet e17b7e024dea47e88c985d53958b4006 (table test-table [id=968827f56c2a4eecb30e3bd252bd0d93]) successfully deleted
I20260924 23:10:08.501706 19471 ts_tablet_manager.cc:1918] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:08.501780 19471 log.cc:1196] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/e17b7e024dea47e88c985d53958b4006
I20260924 23:10:08.502058 19471 ts_tablet_manager.cc:1939] T e17b7e024dea47e88c985d53958b4006 P 080f7adf82c846b0b5b3148b15dc8a50: Deleting consensus metadata
W20260924 23:10:08.502378 18969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46779 (error 108)
I20260924 23:10:08.502488 18968 catalog_manager.cc:4841] TS 080f7adf82c846b0b5b3148b15dc8a50 (127.0.98.2:36105): tablet e17b7e024dea47e88c985d53958b4006 (table test-table [id=968827f56c2a4eecb30e3bd252bd0d93]) successfully deleted
I20260924 23:10:08.502741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19174
I20260924 23:10:08.503363 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:08.502828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46779 (local address 127.0.0.1:41068)
0924 23:10:08.503289 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:08.503337 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
W20260924 23:10:08.503810 18969 catalog_manager.cc:4584] TS 940d641c44094c52abf2d65d0cf06ae6 (127.0.98.1:46779): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e17b7e024dea47e88c985d53958b4006: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46779: connect: Connection refused (error 111)
I20260924 23:10:08.507788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19305
I20260924 23:10:08.512590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 18952
2026-09-24T23:10:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:08 dist-test-slave-ph9j krb5kdc[19476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:08 dist-test-slave-ph9j krb5kdc[19476](info): set up 2 sockets
Sep 24 23:10:08 dist-test-slave-ph9j krb5kdc[19476](info): commencing operation
krb5kdc: starting...
W20260924 23:10:09.557674 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:09 dist-test-slave-ph9j krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:09.676689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:09.758167 19489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:09.758365 19489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:09.758387 19489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:09.759553 19489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:09.759598 19489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:09.759613 19489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:09.759624 19489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:09.759636 19489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:09.761107 19489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:09.761358 19489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:09.761605 19489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:09.766666 19497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:09.766677 19495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:09.767166 19494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:09.767223 19489 server_base.cc:1034] running on GCE node
I20260924 23:10:09.767503 19489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:09.767798 19489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:09.768994 19489 hybrid_clock.cc:648] HybridClock initialized: now 1790291409768968 us; error 43 us; skew 500 ppm
Sep 24 23:10:09 dist-test-slave-ph9j krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:09.778198 19489 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:09.778661 19489 webserver.cc:460] Webserver started at http://127.0.98.62:34077/ using document root <none> and password file <none>
I20260924 23:10:09.778878 19489 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:09.778923 19489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:09.779018 19489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:09.780133 19489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "dd0f871e7bd1446f866b79d3e97e2a5a"
format_stamp: "Formatted at 2026-09-24 23:10:09 on dist-test-slave-ph9j"
server_key: "a215b8b2e5dcc0adc2dbd417d50e3ac9"
server_key_iv: "58e84b7be561c305866b6e0250fb7688"
server_key_version: "encryptionkey@0"
I20260924 23:10:09.780447 19489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd0f871e7bd1446f866b79d3e97e2a5a"
format_stamp: "Formatted at 2026-09-24 23:10:09 on dist-test-slave-ph9j"
server_key: "a215b8b2e5dcc0adc2dbd417d50e3ac9"
server_key_iv: "58e84b7be561c305866b6e0250fb7688"
server_key_version: "encryptionkey@0"
I20260924 23:10:09.781981 19489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:09.782766 19504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:09.782969 19489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:09.783064 19489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "dd0f871e7bd1446f866b79d3e97e2a5a"
format_stamp: "Formatted at 2026-09-24 23:10:09 on dist-test-slave-ph9j"
server_key: "a215b8b2e5dcc0adc2dbd417d50e3ac9"
server_key_iv: "58e84b7be561c305866b6e0250fb7688"
server_key_version: "encryptionkey@0"
I20260924 23:10:09.783155 19489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:09.795794 19489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:09.798316 19489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:09.798429 19489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:09.802465 19489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33555
I20260924 23:10:09.802546 19556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33555 every 8 connection(s)
I20260924 23:10:09.802865 19489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:09.803715 19557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:09.806145 19557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: Bootstrap starting.
I20260924 23:10:09.806667 19557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:09.806895 19557 log.cc:824] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:09.807538 19557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: No bootstrap required, opened a new log
I20260924 23:10:09.809059 19557 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } }
I20260924 23:10:09.809194 19557 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:09.809216 19557 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0f871e7bd1446f866b79d3e97e2a5a, State: Initialized, Role: FOLLOWER
I20260924 23:10:09.809327 19557 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } }
I20260924 23:10:09.809412 19557 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:09.809463 19557 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:09.809537 19557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:09.810139 19557 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } }
I20260924 23:10:09.810254 19557 leader_election.cc:302] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0f871e7bd1446f866b79d3e97e2a5a; no voters:
I20260924 23:10:09.810436 19557 leader_election.cc:288] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:09.810479 19560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:09.810684 19560 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [term 1 LEADER]: Becoming Leader. State: Replica: dd0f871e7bd1446f866b79d3e97e2a5a, State: Running, Role: LEADER
I20260924 23:10:09.810704 19557 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:09.810776 19560 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } }
I20260924 23:10:09.811529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19489
I20260924 23:10:09.811650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:09.811889 392 external_mini_cluster.cc:1428] Setting key 883f9298cff6ea87e8f1fe3dff2410e3
I20260924 23:10:09.811317 19562 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0f871e7bd1446f866b79d3e97e2a5a. Latest consensus state: current_term: 1 leader_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } }
I20260924 23:10:09.812294 19561 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0f871e7bd1446f866b79d3e97e2a5a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33555 } } }
W20260924 23:10:09.812541 19575 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 23:10:09.812583 19561 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:09.812573 19562 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a [sys.catalog]: This master's current role is: LEADER
W20260924 23:10:09.812588 19575 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:09.812827 19579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:09.813015 19579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:09.814656 19579 catalog_manager.cc:1347] Generated new cluster ID: 830ded957b564654adb3edd08b9c230d
I20260924 23:10:09.814708 19579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:10:09 dist-test-slave-ph9j krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:09.822103 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:09.812383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.0.98.62:33555)
0924 23:10:09.812552 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:09.812562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:09.812740 (+ 178us) server_negotiation.cc:407] Connection header received
0924 23:10:09.812960 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:09.812968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:09.813028 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:09.813173 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:09.813955 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:09.815161 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:09.815928 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:09.816374 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:09.816496 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:09.818851 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:09.818855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:09.818856 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:09.818868 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:09.820534 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:09.820894 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:09.820896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:09.820897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:09.820995 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:09.821209 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:09.821210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:09.821210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:09.821464 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:09.821640 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:09.821733 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 23:10:09.821901 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260924 23:10:09.837756 19579 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:09.838521 19579 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:09.847100 19579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: Generated new TSK 0
I20260924 23:10:09.847342 19579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 23:10:09.885620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33555
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:09.966270 19583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:09.966496 19583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:09.966532 19583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:09.967648 19583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:09.967715 19583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:09.967759 19583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:10:09.969233 19583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:09.969499 19583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:09.969725 19583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:09.974689 19589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:09.974689 19588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:09.974697 19591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:09.974905 19583 server_base.cc:1034] running on GCE node
I20260924 23:10:09.975136 19583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:09.975389 19583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:09.976558 19583 hybrid_clock.cc:648] HybridClock initialized: now 1790291409976543 us; error 48 us; skew 500 ppm
Sep 24 23:10:09 dist-test-slave-ph9j krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:09.979218 19583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 23:10:09.979696 19583 webserver.cc:460] Webserver started at http://127.0.98.1:37039/ using document root <none> and password file <none>
I20260924 23:10:09.979921 19583 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:09.979987 19583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:09.980146 19583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:09.981211 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "859034f128cf4e5faa882bfc83d5ebef"
format_stamp: "Formatted at 2026-09-24 23:10:09 on dist-test-slave-ph9j"
server_key: "deb87bc1ec64d9d9261dc73bd4b9c70e"
server_key_iv: "4f181254970c5bbf8385245c778c7eb7"
server_key_version: "encryptionkey@0"
I20260924 23:10:09.981581 19583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "859034f128cf4e5faa882bfc83d5ebef"
format_stamp: "Formatted at 2026-09-24 23:10:09 on dist-test-slave-ph9j"
server_key: "deb87bc1ec64d9d9261dc73bd4b9c70e"
server_key_iv: "4f181254970c5bbf8385245c778c7eb7"
server_key_version: "encryptionkey@0"
I20260924 23:10:09.982952 19583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:09.983748 19598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:09.983927 19583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:09.984050 19583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "859034f128cf4e5faa882bfc83d5ebef"
format_stamp: "Formatted at 2026-09-24 23:10:09 on dist-test-slave-ph9j"
server_key: "deb87bc1ec64d9d9261dc73bd4b9c70e"
server_key_iv: "4f181254970c5bbf8385245c778c7eb7"
server_key_version: "encryptionkey@0"
I20260924 23:10:09.984127 19583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:10.001346 19583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:10.003819 19583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:10.003993 19583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:10.004256 19583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:10.004575 19583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:10.004629 19583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.004676 19583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:10.004700 19583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.010934 19583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41477
I20260924 23:10:10.010980 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41477 every 8 connection(s)
I20260924 23:10:10.011406 19583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 23:10:10 dist-test-slave-ph9j krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:10.020917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19583
I20260924 23:10:10.021071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:10:10.021142 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.012116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55219 (local address 127.0.98.62:33555)
0924 23:10:10.012160 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.012162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.012419 (+ 257us) server_negotiation.cc:407] Connection header received
0924 23:10:10.013415 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.013416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.013434 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.013479 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:10.014219 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.014733 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.015764 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.015969 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.016062 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.018257 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:10.018258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:10.018259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:10.018266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:10.019709 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.020130 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.020131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.020132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.020192 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.020445 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.020446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.020447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.020566 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:10.020645 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.020932 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.021049 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:10.021327 392 external_mini_cluster.cc:1428] Setting key f49251ebc64ef3f30c37ed11fe93ed24
I20260924 23:10:10.021340 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.012095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.98.1:55219)
0924 23:10:10.012260 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.012308 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.013286 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.013599 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.013603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.013858 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:10.014125 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.014130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.014831 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.014832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.015670 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.015672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.016073 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.016074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.016198 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.016731 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:10.016740 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:10.018154 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:10.019811 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:10.019818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:10.019820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:10.020046 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:10.020260 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:10.020261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:10.020261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:10.020376 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:10.020636 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:10.020643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:10.020800 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.020910 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.021062 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260924 23:10:10.021970 19712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33555
I20260924 23:10:10.022078 19712 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:10.022238 19712 heartbeater.cc:511] Master 127.0.98.62:33555 requested a full tablet report, sending...
I20260924 23:10:10.022790 19521 ts_manager.cc:194] Registered new tserver with Master: 859034f128cf4e5faa882bfc83d5ebef (127.0.98.1:41477)
I20260924 23:10:10.023635 19521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55219
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 23:10:10.046649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33555
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:10.128532 19717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:10.128798 19717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:10.128835 19717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:10.130056 19717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:10.130152 19717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:10.130200 19717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:10:10.131729 19717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:10.132019 19717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:10.132265 19717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:10.137634 19722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.137634 19725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.137864 19717 server_base.cc:1034] running on GCE node
W20260924 23:10:10.137636 19723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.138201 19717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:10.138468 19717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:10.139663 19717 hybrid_clock.cc:648] HybridClock initialized: now 1790291410139655 us; error 43 us; skew 500 ppm
Sep 24 23:10:10 dist-test-slave-ph9j krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:10.142627 19717 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 23:10:10.143115 19717 webserver.cc:460] Webserver started at http://127.0.98.2:40737/ using document root <none> and password file <none>
I20260924 23:10:10.143347 19717 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:10.143415 19717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:10.143569 19717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:10.144657 19717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "e705048cc0fd449daac8c0149f3ac281"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "29407038f9f32ada5688e58f50fdc9f7"
server_key_iv: "997a5f594d70af72f9758abb19822286"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.145068 19717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e705048cc0fd449daac8c0149f3ac281"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "29407038f9f32ada5688e58f50fdc9f7"
server_key_iv: "997a5f594d70af72f9758abb19822286"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.146591 19717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:10.147493 19732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.147724 19717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:10.147835 19717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "e705048cc0fd449daac8c0149f3ac281"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "29407038f9f32ada5688e58f50fdc9f7"
server_key_iv: "997a5f594d70af72f9758abb19822286"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.147912 19717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:10.186957 19717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:10.189471 19717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:10.189649 19717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:10.190003 19717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:10.190336 19717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:10.190390 19717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.190434 19717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:10.190460 19717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.196378 19717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39879
I20260924 23:10:10.196465 19845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39879 every 8 connection(s)
I20260924 23:10:10.196801 19717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:10:10.202086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19717
I20260924 23:10:10.202219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:10:10.202435 392 external_mini_cluster.cc:1428] Setting key 036a5a12d3d900f07ca2cfa57ad7e3dd
Sep 24 23:10:10 dist-test-slave-ph9j krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:10.206986 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.197669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54111 (local address 127.0.98.62:33555)
0924 23:10:10.197713 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.197715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.197915 (+ 200us) server_negotiation.cc:407] Connection header received
0924 23:10:10.198853 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.198854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.198865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.198929 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:10.199510 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.200006 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.200997 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.201178 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.201228 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.203809 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:10.203810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:10.203811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:10.203817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:10.205441 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.205857 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.205859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.205859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.205952 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.206221 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.206222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.206222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.206374 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:10.206444 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.206763 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.206865 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:10.207091 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.197617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.98.2:54111)
0924 23:10:10.197772 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.197815 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.198756 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.198986 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.198989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.199246 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:10.199441 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.199445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.200123 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.200124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.200924 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.200926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.201248 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.201248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.201347 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.201938 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:10.201949 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:10.203721 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:10.205555 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:10.205561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:10.205563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:10.205777 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:10.206018 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:10.206018 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:10.206019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:10.206119 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:10.206445 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:10.206451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:10.206602 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.206690 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.206815 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 23:10:10.207617 19846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33555
I20260924 23:10:10.207726 19846 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:10.207908 19846 heartbeater.cc:511] Master 127.0.98.62:33555 requested a full tablet report, sending...
I20260924 23:10:10.208340 19521 ts_manager.cc:194] Registered new tserver with Master: e705048cc0fd449daac8c0149f3ac281 (127.0.98.2:39879)
I20260924 23:10:10.208906 19521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54111
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 23:10:10.228827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33555
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:10.309581 19851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:10.309783 19851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:10.309806 19851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:10.311089 19851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:10.311131 19851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:10.311159 19851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:10:10.312742 19851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:10.313004 19851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:10.313241 19851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:10.318182 19857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.318183 19859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.318224 19856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.318361 19851 server_base.cc:1034] running on GCE node
I20260924 23:10:10.318588 19851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:10.318831 19851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:10.319990 19851 hybrid_clock.cc:648] HybridClock initialized: now 1790291410319969 us; error 39 us; skew 500 ppm
Sep 24 23:10:10 dist-test-slave-ph9j krb5kdc[19476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:10.322559 19851 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 23:10:10.322994 19851 webserver.cc:460] Webserver started at http://127.0.98.3:34303/ using document root <none> and password file <none>
I20260924 23:10:10.323284 19851 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:10.323354 19851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:10.323532 19851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:10.324591 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "f713321afddb41db8812e34be8b2eced"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "cd44a09bf14c96598f75f57a61e41a57"
server_key_iv: "63f5df1cbb8235fcd3f1ba1ef2bc7495"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.324966 19851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f713321afddb41db8812e34be8b2eced"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "cd44a09bf14c96598f75f57a61e41a57"
server_key_iv: "63f5df1cbb8235fcd3f1ba1ef2bc7495"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.326380 19851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:10.327143 19866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.327304 19851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:10.327419 19851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "f713321afddb41db8812e34be8b2eced"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "cd44a09bf14c96598f75f57a61e41a57"
server_key_iv: "63f5df1cbb8235fcd3f1ba1ef2bc7495"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.327493 19851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:10.363062 19851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:10.365175 19851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:10.365343 19851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:10.365592 19851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:10.365955 19851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:10.366009 19851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.366055 19851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:10.366098 19851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.371822 19851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46409
I20260924 23:10:10.371901 19979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46409 every 8 connection(s)
I20260924 23:10:10.372272 19851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:10:10.374044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 19851
I20260924 23:10:10.374156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:10:10.374336 392 external_mini_cluster.cc:1428] Setting key e76e8ab1db66bc73a55fdf504bce307d
Sep 24 23:10:10 dist-test-slave-ph9j krb5kdc[19476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:10.381719 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.373019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40561 (local address 127.0.98.62:33555)
0924 23:10:10.373064 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.373066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.373259 (+ 193us) server_negotiation.cc:407] Connection header received
0924 23:10:10.374343 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.374345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.374358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.374410 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:10.375227 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.375736 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.376741 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.376941 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.377026 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.379122 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:10.379124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:10.379124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:10.379132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:10.380384 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.380789 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.380790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.380791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.380848 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.381095 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.381096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.381096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.381208 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:10.381285 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.381549 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.381625 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:10.381937 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.372962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33555 (local address 127.0.98.3:40561)
0924 23:10:10.373131 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.373173 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.374219 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.374482 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.374486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.374834 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:10.375114 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.375120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.375831 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.375832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.376650 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.376652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.377037 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.377038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.377175 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.377660 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:10.377668 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:10.379030 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:10.380466 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:10.380472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:10.380475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:10.380699 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:10.380916 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:10.380917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:10.380917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:10.381024 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:10.381276 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:10.381282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:10.381420 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.381530 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.381689 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260924 23:10:10.382459 19980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33555
I20260924 23:10:10.382532 19980 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:10.382726 19980 heartbeater.cc:511] Master 127.0.98.62:33555 requested a full tablet report, sending...
I20260924 23:10:10.383071 19521 ts_manager.cc:194] Registered new tserver with Master: f713321afddb41db8812e34be8b2eced (127.0.98.3:46409)
I20260924 23:10:10.383386 19521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40561
I20260924 23:10:10.386022 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:10:10.392789 19578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.387439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.98.62:33555)
0924 23:10:10.387490 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.387491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.387584 (+ 93us) server_negotiation.cc:407] Connection header received
0924 23:10:10.387645 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.387646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.387654 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.387692 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:10.388171 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.388614 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.389321 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.389491 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.389536 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.390767 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:10.390768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:10.390768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:10.390773 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:10.391876 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.392179 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.392179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.392179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.392224 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:10.392351 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:10.392352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:10.392352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:10.392478 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:10.392551 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.392677 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.392732 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:10:10.394210 19521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:10:10.395654 19521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:10:10.401726 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.397704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46409 (local address 127.0.0.1:34972)
0924 23:10:10.397970 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.398017 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.398086 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.398503 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.398506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.398510 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:10.398579 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.398581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.399477 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.399478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.400536 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.400538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.401413 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.401414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.401494 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.401507 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.401581 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.401639 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":68,"threads_started":1}
I20260924 23:10:10.401726 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.397818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.0.98.3:46409)
0924 23:10:10.398325 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.398328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.398337 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:10.398372 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.398374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.398410 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.398497 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:10.398661 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.399363 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.400640 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.401319 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.401410 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.401471 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.401600 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.401638 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":123,"threads_started":1}
I20260924 23:10:10.402328 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.397771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39879 (local address 127.0.0.1:41866)
0924 23:10:10.398021 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.398033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.398088 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.398745 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.398746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.398748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:10.398793 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.398795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.400064 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.400065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.400943 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.400953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.402105 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.402106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.402176 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.402180 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.402210 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.402252 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":142,"threads_started":1}
I20260924 23:10:10.402423 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.398223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41477 (local address 127.0.0.1:34858)
0924 23:10:10.398354 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.398365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.398413 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.398617 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.398617 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.398619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:10.398660 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.398662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.399997 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.399998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.401095 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.401097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.402148 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.402150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.402217 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.402222 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.402273 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.402312 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":22,"thread_start_us":74,"threads_started":1}
I20260924 23:10:10.402505 19914 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc1e289e4234deda4011b5c476a2a81 (DEFAULT_TABLE table=test-table [id=509939eca8054d86b3469f44e3506834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:10.402715 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.398068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34858 (local address 127.0.98.1:41477)
0924 23:10:10.398247 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.398252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.398476 (+ 224us) server_negotiation.cc:407] Connection header received
0924 23:10:10.398520 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.398523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.398539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.398608 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:10.398739 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.399878 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.401175 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.402082 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.402148 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.402208 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.402296 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.402336 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260924 23:10:10.402921 19914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc1e289e4234deda4011b5c476a2a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:10.403069 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.398068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41866 (local address 127.0.98.2:39879)
0924 23:10:10.398248 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.398252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.398260 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:10.398318 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.398321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.398337 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.398873 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:10.398953 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.399982 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.401037 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.401780 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.402541 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.402612 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.402933 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.402986 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260924 23:10:10.403360 19646 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc1e289e4234deda4011b5c476a2a81 (DEFAULT_TABLE table=test-table [id=509939eca8054d86b3469f44e3506834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:10.403671 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc1e289e4234deda4011b5c476a2a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:10.403615 19780 tablet_service.cc:1467] Processing CreateTablet for tablet 9bc1e289e4234deda4011b5c476a2a81 (DEFAULT_TABLE table=test-table [id=509939eca8054d86b3469f44e3506834]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:10.403847 19780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bc1e289e4234deda4011b5c476a2a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:10.406383 19997 tablet_bootstrap.cc:492] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Bootstrap starting.
I20260924 23:10:10.407037 19999 tablet_bootstrap.cc:492] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: Bootstrap starting.
I20260924 23:10:10.407156 19998 tablet_bootstrap.cc:492] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Bootstrap starting.
I20260924 23:10:10.407318 19997 tablet_bootstrap.cc:654] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:10.407712 19999 tablet_bootstrap.cc:654] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:10.407740 19998 tablet_bootstrap.cc:654] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:10.407860 19997 log.cc:824] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:10.408301 19998 log.cc:824] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:10.408327 19999 log.cc:824] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:10.408622 19997 tablet_bootstrap.cc:492] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: No bootstrap required, opened a new log
I20260924 23:10:10.408700 19997 ts_tablet_manager.cc:1397] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:10.409240 19998 tablet_bootstrap.cc:492] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: No bootstrap required, opened a new log
I20260924 23:10:10.409309 19998 ts_tablet_manager.cc:1397] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:10.409333 19999 tablet_bootstrap.cc:492] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: No bootstrap required, opened a new log
I20260924 23:10:10.409392 19999 ts_tablet_manager.cc:1397] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:10.410808 19997 raft_consensus.cc:348] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.411007 19997 raft_consensus.cc:374] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:10.410938 19999 raft_consensus.cc:348] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.411098 19997 raft_consensus.cc:729] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f713321afddb41db8812e34be8b2eced, State: Initialized, Role: FOLLOWER
I20260924 23:10:10.411106 19999 raft_consensus.cc:374] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:10.411141 19999 raft_consensus.cc:729] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859034f128cf4e5faa882bfc83d5ebef, State: Initialized, Role: FOLLOWER
I20260924 23:10:10.411221 19997 consensus_queue.cc:260] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.411283 19999 consensus_queue.cc:260] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.411526 19997 ts_tablet_manager.cc:1428] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:10:10.411410 19998 raft_consensus.cc:348] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.411573 19998 raft_consensus.cc:374] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:10.411615 19998 raft_consensus.cc:729] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e705048cc0fd449daac8c0149f3ac281, State: Initialized, Role: FOLLOWER
I20260924 23:10:10.411662 19980 heartbeater.cc:503] Master 127.0.98.62:33555 was elected leader, sending a full tablet report...
I20260924 23:10:10.411727 19998 consensus_queue.cc:260] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.411592 19999 ts_tablet_manager.cc:1428] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:10.411991 19998 ts_tablet_manager.cc:1428] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 23:10:10.412038 19712 heartbeater.cc:503] Master 127.0.98.62:33555 was elected leader, sending a full tablet report...
I20260924 23:10:10.412051 19846 heartbeater.cc:503] Master 127.0.98.62:33555 was elected leader, sending a full tablet report...
W20260924 23:10:10.447479 19847 tablet.cc:2369] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 23:10:10.512326 19713 tablet.cc:2369] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:10.548749 20004 raft_consensus.cc:482] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:10:10.548866 20004 raft_consensus.cc:504] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.549177 20004 leader_election.cc:288] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f713321afddb41db8812e34be8b2eced (127.0.98.3:46409), e705048cc0fd449daac8c0149f3ac281 (127.0.98.2:39879)
I20260924 23:10:10.552203 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.549474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35269 (local address 127.0.98.2:39879)
0924 23:10:10.549529 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.549532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.549571 (+ 39us) server_negotiation.cc:407] Connection header received
0924 23:10:10.549652 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.549653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.549664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.549733 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:10.549920 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.550410 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.551441 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.551835 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.551919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.551929 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.552099 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.552138 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 23:10:10.552224 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.549352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39879 (local address 127.0.98.1:35269)
0924 23:10:10.549510 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.549522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.549588 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.549738 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.549738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.549741 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:10.549818 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.549820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.550484 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.550485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.551354 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.551357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.551918 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.551919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.551988 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.551992 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.552087 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.552132 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260924 23:10:10.552394 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.549424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39567 (local address 127.0.98.3:46409)
0924 23:10:10.549476 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.549478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.549636 (+ 158us) server_negotiation.cc:407] Connection header received
0924 23:10:10.549731 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.549732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.549742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.549813 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:10.550028 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.550604 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.551619 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.552007 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.552092 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.552102 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.552258 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.552304 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:10:10.552508 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.549352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46409 (local address 127.0.98.1:39567)
0924 23:10:10.549560 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.549573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.549664 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.549829 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.549830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.549832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:10.549939 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.549941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.550698 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.550699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.551522 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.551524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.552093 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.552094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.552170 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.552173 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.552249 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.552420 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260924 23:10:10.552520 19800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc1e289e4234deda4011b5c476a2a81" candidate_uuid: "859034f128cf4e5faa882bfc83d5ebef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e705048cc0fd449daac8c0149f3ac281" is_pre_election: true
I20260924 23:10:10.552672 19800 raft_consensus.cc:2454] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859034f128cf4e5faa882bfc83d5ebef in term 0.
I20260924 23:10:10.552811 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc1e289e4234deda4011b5c476a2a81" candidate_uuid: "859034f128cf4e5faa882bfc83d5ebef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f713321afddb41db8812e34be8b2eced" is_pre_election: true
I20260924 23:10:10.552925 19599 leader_election.cc:302] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859034f128cf4e5faa882bfc83d5ebef, e705048cc0fd449daac8c0149f3ac281; no voters:
I20260924 23:10:10.552956 19934 raft_consensus.cc:2454] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859034f128cf4e5faa882bfc83d5ebef in term 0.
I20260924 23:10:10.553051 20004 raft_consensus.cc:2781] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:10:10.553094 20004 raft_consensus.cc:482] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:10:10.553119 20004 raft_consensus.cc:3037] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:10.553785 20004 raft_consensus.cc:504] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.553941 20004 leader_election.cc:288] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [CANDIDATE]: Term 1 election: Requested vote from peers f713321afddb41db8812e34be8b2eced (127.0.98.3:46409), e705048cc0fd449daac8c0149f3ac281 (127.0.98.2:39879)
I20260924 23:10:10.554121 19934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc1e289e4234deda4011b5c476a2a81" candidate_uuid: "859034f128cf4e5faa882bfc83d5ebef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f713321afddb41db8812e34be8b2eced"
I20260924 23:10:10.554183 19800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bc1e289e4234deda4011b5c476a2a81" candidate_uuid: "859034f128cf4e5faa882bfc83d5ebef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e705048cc0fd449daac8c0149f3ac281"
I20260924 23:10:10.554217 19934 raft_consensus.cc:3037] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:10.554258 19800 raft_consensus.cc:3037] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:10.554947 19800 raft_consensus.cc:2454] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859034f128cf4e5faa882bfc83d5ebef in term 1.
I20260924 23:10:10.555055 19934 raft_consensus.cc:2454] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859034f128cf4e5faa882bfc83d5ebef in term 1.
I20260924 23:10:10.555114 19599 leader_election.cc:302] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859034f128cf4e5faa882bfc83d5ebef, e705048cc0fd449daac8c0149f3ac281; no voters:
I20260924 23:10:10.555207 20004 raft_consensus.cc:2781] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:10.555387 20004 raft_consensus.cc:686] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [term 1 LEADER]: Becoming Leader. State: Replica: 859034f128cf4e5faa882bfc83d5ebef, State: Running, Role: LEADER
I20260924 23:10:10.555481 20004 consensus_queue.cc:237] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } }
I20260924 23:10:10.556299 19521 catalog_manager.cc:5495] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef reported cstate change: term changed from 0 to 1, leader changed from <none> to 859034f128cf4e5faa882bfc83d5ebef (127.0.98.1). New cstate: current_term: 1 leader_uuid: "859034f128cf4e5faa882bfc83d5ebef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859034f128cf4e5faa882bfc83d5ebef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41477 } health_report { overall_health: HEALTHY } } }
I20260924 23:10:10.582262 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.579836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.0.98.1:41477)
0924 23:10:10.579961 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.579963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.579971 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:10.580009 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.580010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.580029 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.580097 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:10.580208 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.580784 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.581475 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.581684 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.581739 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.581864 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:10.582000 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:10.582066 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.582131 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.582150 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:10:10.584270 19934 raft_consensus.cc:1261] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 1 FOLLOWER]: Refusing update from remote peer 859034f128cf4e5faa882bfc83d5ebef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:10.584312 19800 raft_consensus.cc:1261] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 1 FOLLOWER]: Refusing update from remote peer 859034f128cf4e5faa882bfc83d5ebef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:10.584718 20008 consensus_queue.cc:1035] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [LEADER]: Connected to new peer: Peer: permanent_uuid: "e705048cc0fd449daac8c0149f3ac281" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:10.584873 20004 consensus_queue.cc:1035] T 9bc1e289e4234deda4011b5c476a2a81 P 859034f128cf4e5faa882bfc83d5ebef [LEADER]: Connected to new peer: Peer: permanent_uuid: "f713321afddb41db8812e34be8b2eced" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:10.587702 20011 mvcc.cc:205] Tried to move back new op lower bound from 7333033617750028288 to 7333033617635893248. Current Snapshot: MvccSnapshot[applied={T|T < 7333033617750028288}]
I20260924 23:10:10.587997 20013 mvcc.cc:205] Tried to move back new op lower bound from 7333033617750028288 to 7333033617635893248. Current Snapshot: MvccSnapshot[applied={T|T < 7333033617750028288}]
I20260924 23:10:10.589947 19520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:10.590046 19520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:10.590941 19520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd0f871e7bd1446f866b79d3e97e2a5a: Sending DeleteTablet for 3 replicas of tablet 9bc1e289e4234deda4011b5c476a2a81
I20260924 23:10:10.591284 19914 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc1e289e4234deda4011b5c476a2a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34972
I20260924 23:10:10.591316 19646 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc1e289e4234deda4011b5c476a2a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34858
I20260924 23:10:10.591581 20021 tablet_replica.cc:330] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: stopping tablet replica
I20260924 23:10:10.591558 19780 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bc1e289e4234deda4011b5c476a2a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41866
I20260924 23:10:10.591698 20021 raft_consensus.cc:2229] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:10.591929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19583
I20260924 23:10:10.591856 20023 tablet_replica.cc:330] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: stopping tablet replica
I20260924 23:10:10.593204 20023 raft_consensus.cc:2229] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:10.593319 20023 raft_consensus.cc:2258] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:10.593734 20023 ts_tablet_manager.cc:1905] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:10.595858 20023 ts_tablet_manager.cc:1918] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:10.595979 20023 log.cc:1196] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/9bc1e289e4234deda4011b5c476a2a81
I20260924 23:10:10.596285 20023 ts_tablet_manager.cc:1939] T 9bc1e289e4234deda4011b5c476a2a81 P e705048cc0fd449daac8c0149f3ac281: Deleting consensus metadata
I20260924 23:10:10.596760 19505 catalog_manager.cc:4841] TS e705048cc0fd449daac8c0149f3ac281 (127.0.98.2:39879): tablet 9bc1e289e4234deda4011b5c476a2a81 (table test-table [id=509939eca8054d86b3469f44e3506834]) successfully deleted
I20260924 23:10:10.591817 20021 raft_consensus.cc:2258] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 23:10:10.598109 19506 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41477 (error 108)
I20260924 23:10:10.598315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19717
I20260924 23:10:10.598413 20021 ts_tablet_manager.cc:1905] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:10.599182 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.599042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41477 (local address 127.0.0.1:34876)
0924 23:10:10.599107 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.599154 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 23:10:10.599303 19506 catalog_manager.cc:4584] TS 859034f128cf4e5faa882bfc83d5ebef (127.0.98.1:41477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bc1e289e4234deda4011b5c476a2a81: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41477: connect: Connection refused (error 111)
I20260924 23:10:10.600311 20021 ts_tablet_manager.cc:1918] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:10.600373 20021 log.cc:1196] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/9bc1e289e4234deda4011b5c476a2a81
I20260924 23:10:10.600629 20021 ts_tablet_manager.cc:1939] T 9bc1e289e4234deda4011b5c476a2a81 P f713321afddb41db8812e34be8b2eced: Deleting consensus metadata
I20260924 23:10:10.601296 19506 catalog_manager.cc:4841] TS f713321afddb41db8812e34be8b2eced (127.0.98.3:46409): tablet 9bc1e289e4234deda4011b5c476a2a81 (table test-table [id=509939eca8054d86b3469f44e3506834]) successfully deleted
I20260924 23:10:10.604794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19851
I20260924 23:10:10.609247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 19489
2026-09-24T23:10:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2095 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T23:10:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:10Z Disabled control of system clock
I20260924 23:10:10.636528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39565
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:10.717950 20028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:10.718159 20028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:10.718179 20028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:10.719246 20028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:10.719276 20028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:10.719290 20028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:10.719301 20028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:10.719313 20028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:10.720660 20028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:10.720839 20028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:10.721074 20028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:10.725849 20036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.725914 20033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.725870 20034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.726056 20028 server_base.cc:1034] running on GCE node
I20260924 23:10:10.726298 20028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:10.726552 20028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:10.727720 20028 hybrid_clock.cc:648] HybridClock initialized: now 1790291410727690 us; error 42 us; skew 500 ppm
I20260924 23:10:10.728928 20028 webserver.cc:460] Webserver started at http://127.0.98.62:44707/ using document root <none> and password file <none>
I20260924 23:10:10.729180 20028 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:10.729250 20028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:10.729409 20028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:10.730540 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "d4a6abf253be40509f99cf97e85eb4b3"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "36f150137928db38ea129d9d8ca7abfd"
server_key_iv: "206beea2510e9abac7e64c11e744bea1"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.730906 20028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4a6abf253be40509f99cf97e85eb4b3"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "36f150137928db38ea129d9d8ca7abfd"
server_key_iv: "206beea2510e9abac7e64c11e744bea1"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.732173 20028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:10.732923 20042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.733111 20028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:10.733215 20028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "d4a6abf253be40509f99cf97e85eb4b3"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "36f150137928db38ea129d9d8ca7abfd"
server_key_iv: "206beea2510e9abac7e64c11e744bea1"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.733310 20028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:10.749759 20028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:10.750170 20028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:10.750334 20028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:10.754240 20028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39565
I20260924 23:10:10.754274 20094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39565 every 8 connection(s)
I20260924 23:10:10.754608 20028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:10.755573 20095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:10.757777 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3: Bootstrap starting.
I20260924 23:10:10.758380 20095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:10.758697 20095 log.cc:824] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:10.759352 20095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3: No bootstrap required, opened a new log
I20260924 23:10:10.760623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20028
I20260924 23:10:10.760710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:10.760915 392 external_mini_cluster.cc:1428] Setting key 1cdb7a395302f112c038b7b7a68d81d7
I20260924 23:10:10.761106 20095 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a6abf253be40509f99cf97e85eb4b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260924 23:10:10.761267 20095 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:10.761302 20095 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a6abf253be40509f99cf97e85eb4b3, State: Initialized, Role: FOLLOWER
I20260924 23:10:10.761433 20095 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a6abf253be40509f99cf97e85eb4b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260924 23:10:10.761518 20095 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:10.761560 20095 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:10.761610 20095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:10.762290 20095 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a6abf253be40509f99cf97e85eb4b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260924 23:10:10.762403 20095 leader_election.cc:302] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a6abf253be40509f99cf97e85eb4b3; no voters:
I20260924 23:10:10.762568 20095 leader_election.cc:288] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:10.762636 20100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:10.762794 20100 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [term 1 LEADER]: Becoming Leader. State: Replica: d4a6abf253be40509f99cf97e85eb4b3, State: Running, Role: LEADER
I20260924 23:10:10.762827 20095 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:10.762908 20100 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a6abf253be40509f99cf97e85eb4b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } }
I20260924 23:10:10.763307 20100 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a6abf253be40509f99cf97e85eb4b3. Latest consensus state: current_term: 1 leader_uuid: "d4a6abf253be40509f99cf97e85eb4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a6abf253be40509f99cf97e85eb4b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } } }
I20260924 23:10:10.763384 20100 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:10.763319 20101 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a6abf253be40509f99cf97e85eb4b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a6abf253be40509f99cf97e85eb4b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39565 } } }
I20260924 23:10:10.763425 20101 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:10.763590 20107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:10.764057 20107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:10.767151 20107 catalog_manager.cc:1347] Generated new cluster ID: a89435e6b861411bb3167b9e6471e388
I20260924 23:10:10.767206 20107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:10.767805 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.761451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40552 (local address 127.0.98.62:39565)
0924 23:10:10.761715 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.761724 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.761788 (+ 64us) server_negotiation.cc:407] Connection header received
0924 23:10:10.761996 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.762001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.762064 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.762174 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:10.762437 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.764242 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.765344 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.765987 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.766106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.767132 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:10.767136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:10.767136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:10.767345 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:10.767377 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:10.767388 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:10.767475 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.767550 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.767628 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260924 23:10:10.775519 20107 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:10.776039 20107 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:10.778795 20107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3: Generated new TSK 0
I20260924 23:10:10.778965 20107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:10.780917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39565
--builtin_ntp_servers=127.0.98.20:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 23:10:10.862462 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:10.862718 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:10.862756 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:10.864019 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:10.864078 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:10.864121 20119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 23:10:10.865569 20119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:10.865839 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:10.866164 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:10.872143 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.872153 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.872391 20119 server_base.cc:1034] running on GCE node
W20260924 23:10:10.872143 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.872723 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:10.873013 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:10.874188 20119 hybrid_clock.cc:648] HybridClock initialized: now 1790291410874175 us; error 32 us; skew 500 ppm
I20260924 23:10:10.875737 20119 webserver.cc:460] Webserver started at http://127.0.98.1:36027/ using document root <none> and password file <none>
I20260924 23:10:10.876030 20119 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:10.876101 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:10.876240 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:10.877696 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data/instance:
uuid: "96f80d84a0b24b109ca7afdd9a7c1e10"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "397711ff5cb8d2606ed1eddeb405dd0d"
server_key_iv: "5941d994ff48b212bd7fa84323a13acc"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.878139 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96f80d84a0b24b109ca7afdd9a7c1e10"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "397711ff5cb8d2606ed1eddeb405dd0d"
server_key_iv: "5941d994ff48b212bd7fa84323a13acc"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.879879 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:10.880759 20133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.880951 20119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:10.881062 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal
uuid: "96f80d84a0b24b109ca7afdd9a7c1e10"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "397711ff5cb8d2606ed1eddeb405dd0d"
server_key_iv: "5941d994ff48b212bd7fa84323a13acc"
server_key_version: "encryptionkey@0"
I20260924 23:10:10.881137 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:10.898124 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:10.898439 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:10.898594 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:10.898834 20119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:10.899178 20119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:10.899216 20119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.899240 20119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:10.899252 20119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:10.904845 20119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38133
I20260924 23:10:10.904927 20246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38133 every 8 connection(s)
I20260924 23:10:10.905273 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/data/info.pb
I20260924 23:10:10.905958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20119
I20260924 23:10:10.906068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-0/wal/instance
I20260924 23:10:10.906311 392 external_mini_cluster.cc:1428] Setting key 135d3bd57692f84a44fbc7f49e2ff727
I20260924 23:10:10.907308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39565
--builtin_ntp_servers=127.0.98.20:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:10:10.911229 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.906170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48351 (local address 127.0.98.62:39565)
0924 23:10:10.906228 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:10.906230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:10.906427 (+ 197us) server_negotiation.cc:407] Connection header received
0924 23:10:10.907073 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:10.907073 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:10.907088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:10.907151 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:10.907989 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.908618 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.909593 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.909967 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.910042 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.910903 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:10.910904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:10.910904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:10.910910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:10.910929 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:10.910934 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:10.910999 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:10.911071 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 23:10:10.911112 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 23:10:10.911332 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:10.906047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39565 (local address 127.0.98.1:48351)
0924 23:10:10.906281 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:10.906321 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:10.906950 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:10.907227 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:10.907231 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:10.907241 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:10.907893 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.907899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.908717 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.908718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.909503 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:10.909506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:10.910053 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:10.910055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:10.910169 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:10.910763 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:10.910763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:10.910791 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:10.910814 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:10.911003 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:10.911008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:10.911010 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:10:10.911069 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 23:10:10.911149 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260924 23:10:10.911823 20247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39565
I20260924 23:10:10.911929 20247 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:10.912114 20247 heartbeater.cc:511] Master 127.0.98.62:39565 requested a full tablet report, sending...
I20260924 23:10:10.912680 20059 ts_manager.cc:194] Registered new tserver with Master: 96f80d84a0b24b109ca7afdd9a7c1e10 (127.0.98.1:38133)
I20260924 23:10:10.913678 20059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48351
W20260924 23:10:10.988842 20250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:10.989042 20250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:10.989063 20250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:10.990214 20250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:10.990253 20250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:10.990278 20250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 23:10:10.991555 20250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:10.991788 20250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:10.992033 20250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:10.997054 20256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.997040 20258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:10.997041 20255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:10.997398 20250 server_base.cc:1034] running on GCE node
I20260924 23:10:10.997594 20250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:10.997793 20250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:10.998925 20250 hybrid_clock.cc:648] HybridClock initialized: now 1790291410998912 us; error 30 us; skew 500 ppm
I20260924 23:10:11.000085 20250 webserver.cc:460] Webserver started at http://127.0.98.2:35443/ using document root <none> and password file <none>
I20260924 23:10:11.000274 20250 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:11.000315 20250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:11.000406 20250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:11.001432 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data/instance:
uuid: "efb326446f8d4276861492c53d445f0f"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "bfa945d1e6d646880f14a1f6b1668ae4"
server_key_iv: "921bb2bccf5ad7e0c1f467a5f21fff70"
server_key_version: "encryptionkey@0"
I20260924 23:10:11.001729 20250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal/instance:
uuid: "efb326446f8d4276861492c53d445f0f"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "bfa945d1e6d646880f14a1f6b1668ae4"
server_key_iv: "921bb2bccf5ad7e0c1f467a5f21fff70"
server_key_version: "encryptionkey@0"
I20260924 23:10:11.003111 20250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:11.003786 20264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:11.003959 20250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:11.004026 20250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal
uuid: "efb326446f8d4276861492c53d445f0f"
format_stamp: "Formatted at 2026-09-24 23:10:10 on dist-test-slave-ph9j"
server_key: "bfa945d1e6d646880f14a1f6b1668ae4"
server_key_iv: "921bb2bccf5ad7e0c1f467a5f21fff70"
server_key_version: "encryptionkey@0"
I20260924 23:10:11.004093 20250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:11.023272 20250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:11.023550 20250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:11.023655 20250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:11.023847 20250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:11.024199 20250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:11.024236 20250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:11.024258 20250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:11.024271 20250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:11.030386 20250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40829
I20260924 23:10:11.030428 20377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40829 every 8 connection(s)
I20260924 23:10:11.030786 20250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/data/info.pb
I20260924 23:10:11.032428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20250
I20260924 23:10:11.032559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal/instance
I20260924 23:10:11.032796 392 external_mini_cluster.cc:1428] Setting key 95836ffbccfc6ca2253e8bdc9b4ca0ce
I20260924 23:10:11.033811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39565
--builtin_ntp_servers=127.0.98.20:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 23:10:11.036727 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.031569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40561 (local address 127.0.98.62:39565)
0924 23:10:11.031616 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.031618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.031893 (+ 275us) server_negotiation.cc:407] Connection header received
0924 23:10:11.032377 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.032379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.032394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.032476 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:11.033338 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.034098 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.035093 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.035376 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.035451 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.036325 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:11.036326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:11.036327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:11.036332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:11.036349 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:11.036354 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:11.036504 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.036621 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.036650 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:10:11.036947 20380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.031539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39565 (local address 127.0.98.2:40561)
0924 23:10:11.031733 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.031774 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.032263 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.032605 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.032611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.032621 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:11.033261 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.033267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.034184 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.034185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.034984 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.034986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.035460 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.035461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.035611 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.036169 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:11.036169 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:11.036198 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:11.036220 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:11.036506 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:11.036520 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:11.036522 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.036617 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.036716 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260924 23:10:11.037397 20378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39565
I20260924 23:10:11.037504 20378 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:11.037704 20378 heartbeater.cc:511] Master 127.0.98.62:39565 requested a full tablet report, sending...
I20260924 23:10:11.038120 20059 ts_manager.cc:194] Registered new tserver with Master: efb326446f8d4276861492c53d445f0f (127.0.98.2:40829)
I20260924 23:10:11.038659 20059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40561
W20260924 23:10:11.113014 20381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:11.113224 20381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:11.113245 20381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:11.114351 20381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:11.114383 20381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:11.114408 20381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 23:10:11.115645 20381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:11.115819 20381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:11.116057 20381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:11.121205 20387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:11.121214 20386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:11.121214 20389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:11.121317 20381 server_base.cc:1034] running on GCE node
I20260924 23:10:11.121644 20381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:11.121888 20381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:11.123098 20381 hybrid_clock.cc:648] HybridClock initialized: now 1790291411123076 us; error 39 us; skew 500 ppm
I20260924 23:10:11.124358 20381 webserver.cc:460] Webserver started at http://127.0.98.3:39777/ using document root <none> and password file <none>
I20260924 23:10:11.124560 20381 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:11.124603 20381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:11.124693 20381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:11.125725 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b23ce727ca141998d822d4a6aafd389"
format_stamp: "Formatted at 2026-09-24 23:10:11 on dist-test-slave-ph9j"
server_key: "a94e40acdee6140f47ae889dc2aa8510"
server_key_iv: "7d6a61d9c319ee12c801d8d3d50ced46"
server_key_version: "encryptionkey@0"
I20260924 23:10:11.126119 20381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b23ce727ca141998d822d4a6aafd389"
format_stamp: "Formatted at 2026-09-24 23:10:11 on dist-test-slave-ph9j"
server_key: "a94e40acdee6140f47ae889dc2aa8510"
server_key_iv: "7d6a61d9c319ee12c801d8d3d50ced46"
server_key_version: "encryptionkey@0"
I20260924 23:10:11.127441 20381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:11.128366 20395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:11.128543 20381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:11.128638 20381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal
uuid: "3b23ce727ca141998d822d4a6aafd389"
format_stamp: "Formatted at 2026-09-24 23:10:11 on dist-test-slave-ph9j"
server_key: "a94e40acdee6140f47ae889dc2aa8510"
server_key_iv: "7d6a61d9c319ee12c801d8d3d50ced46"
server_key_version: "encryptionkey@0"
I20260924 23:10:11.128714 20381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:11.152328 20381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:11.152657 20381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:11.152817 20381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:11.153079 20381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 23:10:11.153427 20381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 23:10:11.153462 20381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:11.153486 20381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 23:10:11.153499 20381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:11.159617 20381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45511
I20260924 23:10:11.159662 20508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45511 every 8 connection(s)
I20260924 23:10:11.160005 20381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/data/info.pb
I20260924 23:10:11.165308 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.160771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34251 (local address 127.0.98.62:39565)
0924 23:10:11.160838 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.160840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.161069 (+ 229us) server_negotiation.cc:407] Connection header received
0924 23:10:11.161522 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.161523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.161534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.161598 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:11.162433 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.162922 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.163979 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.164173 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.164225 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.165030 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:11.165031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:11.165031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:11.165035 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:11.165062 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:11.165067 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:11.165110 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.165203 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.165226 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:10:11.165525 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.160694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39565 (local address 127.0.98.3:34251)
0924 23:10:11.160902 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.160944 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.161440 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.161672 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.161676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.161684 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:11.162355 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.162362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.162996 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.162997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.163909 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.163912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.164246 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.164247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.164355 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.164907 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 23:10:11.164908 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:11.164935 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 23:10:11.164956 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:11.165127 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:11.165132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:11.165134 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.165192 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.165285 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260924 23:10:11.166170 20509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39565
I20260924 23:10:11.166299 20509 heartbeater.cc:464] Registering TS with master...
I20260924 23:10:11.166517 20509 heartbeater.cc:511] Master 127.0.98.62:39565 requested a full tablet report, sending...
I20260924 23:10:11.166906 20059 ts_manager.cc:194] Registered new tserver with Master: 3b23ce727ca141998d822d4a6aafd389 (127.0.98.3:45511)
I20260924 23:10:11.167305 20059 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34251
I20260924 23:10:11.168798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20381
I20260924 23:10:11.168915 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal/instance
I20260924 23:10:11.169178 392 external_mini_cluster.cc:1428] Setting key 83646a86f4cc3e256d84a2b7e880af3a
I20260924 23:10:11.170090 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 23:10:11.174291 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.171380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40566 (local address 127.0.98.62:39565)
0924 23:10:11.171429 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.171431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.171496 (+ 65us) server_negotiation.cc:407] Connection header received
0924 23:10:11.171573 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.171573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.171586 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.171641 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:11.171816 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.172298 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.173059 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.173262 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.173336 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.174020 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:11.174021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:11.174022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:11.174026 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:11.174046 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:11.174051 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:11.174121 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.174181 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.174204 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 23:10:11.175784 20059 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 23:10:11.177326 20059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 23:10:11.183559 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.179502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38133 (local address 127.0.0.1:48146)
0924 23:10:11.179689 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.179729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.179789 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.180383 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.180384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.180386 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:11.180425 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.180427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.181659 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.181660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.182597 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.182599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.183278 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.183280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.183352 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.183364 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.183437 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.183485 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260924 23:10:11.183580 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.179606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.0.98.1:38133)
0924 23:10:11.180221 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.180226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.180235 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:11.180270 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.180272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.180289 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.180494 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:11.180570 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.181577 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.182686 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.183183 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.183269 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.183364 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.183452 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.183491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":91,"threads_started":1}
I20260924 23:10:11.183916 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.179714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40829 (local address 127.0.0.1:53066)
0924 23:10:11.179917 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.179927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.179969 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.180367 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.180368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.180370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:11.180412 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.180413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.181264 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.181265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.182649 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.182651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.183673 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.183674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.183748 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.183753 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.183808 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.183844 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":38,"thread_start_us":69,"threads_started":1}
I20260924 23:10:11.184190 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.179502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45511 (local address 127.0.0.1:54052)
0924 23:10:11.179832 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.179841 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.179914 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.180106 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.180109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.180111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:11.180172 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.180174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.181432 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.181433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.182648 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.182651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.183956 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.183957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.184060 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.184065 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.184106 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.184140 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":28,"thread_start_us":76,"threads_started":1}
I20260924 23:10:11.184325 20181 tablet_service.cc:1467] Processing CreateTablet for tablet bcd528954bad4688aad15ccc672d5af5 (DEFAULT_TABLE table=test-table [id=b98a50df98624165b6d7db829b3d4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:11.184485 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.179605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.0.98.3:45511)
0924 23:10:11.179945 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.179951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.179962 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:10:11.180002 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.180004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.180020 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.180089 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:11.180255 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.181352 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.182752 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.183480 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.183544 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.184317 (+ 773us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.184343 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.184382 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":86,"threads_started":1}
I20260924 23:10:11.184713 20181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd528954bad4688aad15ccc672d5af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:11.185245 20443 tablet_service.cc:1467] Processing CreateTablet for tablet bcd528954bad4688aad15ccc672d5af5 (DEFAULT_TABLE table=test-table [id=b98a50df98624165b6d7db829b3d4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:11.185582 20443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd528954bad4688aad15ccc672d5af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:11.188258 20523 tablet_bootstrap.cc:492] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: Bootstrap starting.
I20260924 23:10:11.188349 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.179932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.0.98.2:40829)
0924 23:10:11.180087 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.180092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.180101 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:11.180142 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.180144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.180229 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.180304 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:11.180495 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.181177 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.182752 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.183472 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.183551 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.183632 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.183840 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.188259 (+ 4419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260924 23:10:11.188947 20524 tablet_bootstrap.cc:492] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Bootstrap starting.
I20260924 23:10:11.189026 20312 tablet_service.cc:1467] Processing CreateTablet for tablet bcd528954bad4688aad15ccc672d5af5 (DEFAULT_TABLE table=test-table [id=b98a50df98624165b6d7db829b3d4485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 23:10:11.189209 20523 tablet_bootstrap.cc:654] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:11.189411 20312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcd528954bad4688aad15ccc672d5af5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:11.189527 20523 log.cc:824] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:11.189612 20524 tablet_bootstrap.cc:654] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:11.189952 20524 log.cc:824] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:11.190414 20523 tablet_bootstrap.cc:492] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: No bootstrap required, opened a new log
I20260924 23:10:11.190490 20523 ts_tablet_manager.cc:1397] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:11.190676 20524 tablet_bootstrap.cc:492] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: No bootstrap required, opened a new log
I20260924 23:10:11.190753 20524 ts_tablet_manager.cc:1397] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:11.192492 20524 raft_consensus.cc:348] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.192646 20524 raft_consensus.cc:374] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:11.192689 20524 raft_consensus.cc:729] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b23ce727ca141998d822d4a6aafd389, State: Initialized, Role: FOLLOWER
I20260924 23:10:11.192799 20524 consensus_queue.cc:260] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.193023 20527 tablet_bootstrap.cc:492] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Bootstrap starting.
I20260924 23:10:11.193193 20524 ts_tablet_manager.cc:1428] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:10:11.193318 20509 heartbeater.cc:503] Master 127.0.98.62:39565 was elected leader, sending a full tablet report...
I20260924 23:10:11.193475 20523 raft_consensus.cc:348] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.193632 20523 raft_consensus.cc:374] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:11.193679 20523 raft_consensus.cc:729] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f80d84a0b24b109ca7afdd9a7c1e10, State: Initialized, Role: FOLLOWER
I20260924 23:10:11.193697 20527 tablet_bootstrap.cc:654] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:11.193800 20523 consensus_queue.cc:260] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.194031 20527 log.cc:824] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:11.194132 20523 ts_tablet_manager.cc:1428] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260924 23:10:11.194252 20247 heartbeater.cc:503] Master 127.0.98.62:39565 was elected leader, sending a full tablet report...
I20260924 23:10:11.194774 20527 tablet_bootstrap.cc:492] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: No bootstrap required, opened a new log
I20260924 23:10:11.194844 20527 ts_tablet_manager.cc:1397] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 23:10:11.196781 20527 raft_consensus.cc:348] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.196903 20527 raft_consensus.cc:374] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:11.196924 20527 raft_consensus.cc:729] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb326446f8d4276861492c53d445f0f, State: Initialized, Role: FOLLOWER
I20260924 23:10:11.197042 20527 consensus_queue.cc:260] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.197325 20527 ts_tablet_manager.cc:1428] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 23:10:11.197395 20378 heartbeater.cc:503] Master 127.0.98.62:39565 was elected leader, sending a full tablet report...
W20260924 23:10:11.281330 20379 tablet.cc:2369] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:11.368403 20529 raft_consensus.cc:482] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 23:10:11.368587 20529 raft_consensus.cc:504] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.368979 20529 leader_election.cc:288] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b23ce727ca141998d822d4a6aafd389 (127.0.98.3:45511), efb326446f8d4276861492c53d445f0f (127.0.98.2:40829)
I20260924 23:10:11.372233 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.369342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34471 (local address 127.0.98.3:45511)
0924 23:10:11.369395 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.369398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.369409 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:10:11.369469 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.369470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.369487 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.369575 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:11.369744 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.370295 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.371348 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.371869 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.371952 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.371963 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.372121 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.372160 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 23:10:11.372287 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.369206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45511 (local address 127.0.98.1:34471)
0924 23:10:11.369269 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.369283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.369329 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.369584 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.369585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.369587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:11.369667 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.369669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.370383 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.370384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.371247 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.371250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.371958 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.371959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.372033 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.372036 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.372111 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.372155 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 23:10:11.372294 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.369424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34169 (local address 127.0.98.2:40829)
0924 23:10:11.369501 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.369503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.369617 (+ 114us) server_negotiation.cc:407] Connection header received
0924 23:10:11.369692 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.369693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.369702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.369772 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 23:10:11.369968 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.370497 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.371503 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.371902 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.371984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.371994 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.372171 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.372212 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 23:10:11.372303 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.369340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40829 (local address 127.0.98.1:34169)
0924 23:10:11.369542 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.369550 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:11.369626 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:11.369777 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:11.369778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:11.369780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 23:10:11.369865 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.369867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.370597 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.370599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.371385 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:11.371387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.371988 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.371989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:11.372089 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.372092 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 23:10:11.372164 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:10:11.372202 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":127,"threads_started":1}
I20260924 23:10:11.372605 20332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd528954bad4688aad15ccc672d5af5" candidate_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb326446f8d4276861492c53d445f0f" is_pre_election: true
I20260924 23:10:11.372792 20332 raft_consensus.cc:2454] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f80d84a0b24b109ca7afdd9a7c1e10 in term 0.
I20260924 23:10:11.373026 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd528954bad4688aad15ccc672d5af5" candidate_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b23ce727ca141998d822d4a6aafd389" is_pre_election: true
I20260924 23:10:11.373062 20134 leader_election.cc:302] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f80d84a0b24b109ca7afdd9a7c1e10, efb326446f8d4276861492c53d445f0f; no voters:
I20260924 23:10:11.373157 20463 raft_consensus.cc:2454] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96f80d84a0b24b109ca7afdd9a7c1e10 in term 0.
I20260924 23:10:11.373190 20529 raft_consensus.cc:2781] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 23:10:11.373276 20529 raft_consensus.cc:482] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 23:10:11.373310 20529 raft_consensus.cc:3037] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:11.374155 20529 raft_consensus.cc:504] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.374293 20529 leader_election.cc:288] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [CANDIDATE]: Term 1 election: Requested vote from peers 3b23ce727ca141998d822d4a6aafd389 (127.0.98.3:45511), efb326446f8d4276861492c53d445f0f (127.0.98.2:40829)
I20260924 23:10:11.374445 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd528954bad4688aad15ccc672d5af5" candidate_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b23ce727ca141998d822d4a6aafd389"
I20260924 23:10:11.374480 20332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcd528954bad4688aad15ccc672d5af5" candidate_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb326446f8d4276861492c53d445f0f"
I20260924 23:10:11.374516 20463 raft_consensus.cc:3037] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:11.374596 20332 raft_consensus.cc:3037] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:11.375288 20463 raft_consensus.cc:2454] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f80d84a0b24b109ca7afdd9a7c1e10 in term 1.
I20260924 23:10:11.375303 20332 raft_consensus.cc:2454] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96f80d84a0b24b109ca7afdd9a7c1e10 in term 1.
I20260924 23:10:11.375473 20134 leader_election.cc:302] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96f80d84a0b24b109ca7afdd9a7c1e10, efb326446f8d4276861492c53d445f0f; no voters:
I20260924 23:10:11.375602 20529 raft_consensus.cc:2781] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:11.375787 20529 raft_consensus.cc:686] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [term 1 LEADER]: Becoming Leader. State: Replica: 96f80d84a0b24b109ca7afdd9a7c1e10, State: Running, Role: LEADER
I20260924 23:10:11.375895 20529 consensus_queue.cc:237] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } }
I20260924 23:10:11.376794 20058 catalog_manager.cc:5495] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96f80d84a0b24b109ca7afdd9a7c1e10 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96f80d84a0b24b109ca7afdd9a7c1e10" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38133 } health_report { overall_health: HEALTHY } } }
W20260924 23:10:11.406047 20248 tablet.cc:2369] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:11.409440 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.406796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.0.98.1:38133)
0924 23:10:11.406874 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:11.406877 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:11.406885 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:11.406932 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:11.406933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:11.406952 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:11.407035 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:11.407188 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.407711 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.408483 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:11.408802 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:11.408881 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:11.409067 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:11.409181 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:11.409279 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:11.409299 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 23:10:11.409320 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 23:10:11.410513 20510 tablet.cc:2369] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:11.411409 20463 raft_consensus.cc:1261] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 1 FOLLOWER]: Refusing update from remote peer 96f80d84a0b24b109ca7afdd9a7c1e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:11.411412 20332 raft_consensus.cc:1261] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 1 FOLLOWER]: Refusing update from remote peer 96f80d84a0b24b109ca7afdd9a7c1e10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:11.411741 20529 consensus_queue.cc:1035] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b23ce727ca141998d822d4a6aafd389" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:11.411906 20533 consensus_queue.cc:1035] T bcd528954bad4688aad15ccc672d5af5 P 96f80d84a0b24b109ca7afdd9a7c1e10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb326446f8d4276861492c53d445f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:11.414439 20538 mvcc.cc:205] Tried to move back new op lower bound from 7333033621138108416 to 7333033620996476928. Current Snapshot: MvccSnapshot[applied={T|T < 7333033621138108416}]
I20260924 23:10:11.417328 20058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:11.417444 20058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 23:10:11.418556 20058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4a6abf253be40509f99cf97e85eb4b3: Sending DeleteTablet for 3 replicas of tablet bcd528954bad4688aad15ccc672d5af5
I20260924 23:10:11.418884 20443 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd528954bad4688aad15ccc672d5af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:11 UTC) from {username='slave'} at 127.0.0.1:54052
I20260924 23:10:11.418898 20312 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd528954bad4688aad15ccc672d5af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:11 UTC) from {username='slave'} at 127.0.0.1:53066
I20260924 23:10:11.418998 20181 tablet_service.cc:1514] Processing DeleteTablet for tablet bcd528954bad4688aad15ccc672d5af5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 23:10:11 UTC) from {username='slave'} at 127.0.0.1:48146
I20260924 23:10:11.419181 20546 tablet_replica.cc:330] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: stopping tablet replica
I20260924 23:10:11.419201 20547 tablet_replica.cc:330] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: stopping tablet replica
I20260924 23:10:11.419332 20546 raft_consensus.cc:2229] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:11.419384 20547 raft_consensus.cc:2229] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 23:10:11.419447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20119
I20260924 23:10:11.420158 20547 raft_consensus.cc:2258] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:11.420650 20547 ts_tablet_manager.cc:1905] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:11.421321 20546 raft_consensus.cc:2258] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 23:10:11.421778 20546 ts_tablet_manager.cc:1905] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 23:10:11.422191 20547 ts_tablet_manager.cc:1918] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:11.422279 20547 log.cc:1196] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-1/wal/wals/bcd528954bad4688aad15ccc672d5af5
I20260924 23:10:11.422524 20547 ts_tablet_manager.cc:1939] T bcd528954bad4688aad15ccc672d5af5 P efb326446f8d4276861492c53d445f0f: Deleting consensus metadata
I20260924 23:10:11.422922 20043 catalog_manager.cc:4841] TS efb326446f8d4276861492c53d445f0f (127.0.98.2:40829): tablet bcd528954bad4688aad15ccc672d5af5 (table test-table [id=b98a50df98624165b6d7db829b3d4485]) successfully deleted
I20260924 23:10:11.423740 20546 ts_tablet_manager.cc:1918] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 23:10:11.423815 20546 log.cc:1196] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Deleting WAL directory at /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790291169445806-392-0/minicluster-data/ts-2/wal/wals/bcd528954bad4688aad15ccc672d5af5
I20260924 23:10:11.424043 20546 ts_tablet_manager.cc:1939] T bcd528954bad4688aad15ccc672d5af5 P 3b23ce727ca141998d822d4a6aafd389: Deleting consensus metadata
I20260924 23:10:11.424450 20043 catalog_manager.cc:4841] TS 3b23ce727ca141998d822d4a6aafd389 (127.0.98.3:45511): tablet bcd528954bad4688aad15ccc672d5af5 (table test-table [id=b98a50df98624165b6d7db829b3d4485]) successfully deleted
W20260924 23:10:11.425388 20044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38133 (error 108)
I20260924 23:10:11.425868 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:11.425750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38133 (local address 127.0.0.1:48158)
0924 23:10:11.425803 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:11.425840 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:10:11.425978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20250
W20260924 23:10:11.426034 20044 catalog_manager.cc:4584] TS 96f80d84a0b24b109ca7afdd9a7c1e10 (127.0.98.1:38133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcd528954bad4688aad15ccc672d5af5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38133: connect: Connection refused (error 111)
I20260924 23:10:11.430862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20381
I20260924 23:10:11.435418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20028
2026-09-24T23:10:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (825 ms)
[----------] 6 tests from EncryptionPolicyTest (8598 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:11 dist-test-slave-ph9j krb5kdc[20552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:11 dist-test-slave-ph9j krb5kdc[20552](info): set up 2 sockets
Sep 24 23:10:11 dist-test-slave-ph9j krb5kdc[20552](info): commencing operation
krb5kdc: starting...
W20260924 23:10:12.469784 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:12 dist-test-slave-ph9j krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:12.595206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37907
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:12.676107 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:12.676316 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:12.676338 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:12.677388 20565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:12.677423 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:12.677438 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:12.677450 20565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:12.677462 20565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:12.678854 20565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37907
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:12.679037 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:12.679198 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:12.683964 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:12.683972 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:12.684163 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:12.684257 20565 server_base.cc:1034] running on GCE node
I20260924 23:10:12.684653 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:12.685007 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:12.686156 20565 hybrid_clock.cc:648] HybridClock initialized: now 1790291412686138 us; error 34 us; skew 500 ppm
Sep 24 23:10:12 dist-test-slave-ph9j krb5kdc[20552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:12.688642 20565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:12.689114 20565 webserver.cc:460] Webserver started at http://127.0.0.1:41943/ using document root <none> and password file <none>
I20260924 23:10:12.689337 20565 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:12.689404 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:12.689549 20565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:12.690675 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf"
format_stamp: "Formatted at 2026-09-24 23:10:12 on dist-test-slave-ph9j"
server_key: "8131f2110a22a30606da473de3fc60de"
server_key_iv: "be6bb1f524de8e8f09ff1166b88fa5e1"
server_key_version: "encryptionkey@0"
I20260924 23:10:12.691033 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf"
format_stamp: "Formatted at 2026-09-24 23:10:12 on dist-test-slave-ph9j"
server_key: "8131f2110a22a30606da473de3fc60de"
server_key_iv: "be6bb1f524de8e8f09ff1166b88fa5e1"
server_key_version: "encryptionkey@0"
I20260924 23:10:12.692349 20565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:10:12.693126 20580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:12.693300 20565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:12.693401 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf"
format_stamp: "Formatted at 2026-09-24 23:10:12 on dist-test-slave-ph9j"
server_key: "8131f2110a22a30606da473de3fc60de"
server_key_iv: "be6bb1f524de8e8f09ff1166b88fa5e1"
server_key_version: "encryptionkey@0"
I20260924 23:10:12.693472 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:12.720252 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:12.722203 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:12.722369 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:12.726266 20565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37907
I20260924 23:10:12.726327 20632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37907 every 8 connection(s)
I20260924 23:10:12.726689 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:12.727543 20633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:12.729727 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf: Bootstrap starting.
I20260924 23:10:12.730386 20633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:12.730502 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20565
I20260924 23:10:12.730640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:12.730777 20633 log.cc:824] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:12.730859 392 external_mini_cluster.cc:1428] Setting key ab1bd83b2008892c2cf06d17c9d64af4
I20260924 23:10:12.731639 20633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf: No bootstrap required, opened a new log
I20260924 23:10:12.734369 20633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260924 23:10:12.734555 20633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:12.734591 20633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748d3ede2acb4fdf92c751bdb2bf1bdf, State: Initialized, Role: FOLLOWER
I20260924 23:10:12.734712 20633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260924 23:10:12.734781 20633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:12.734808 20633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:12.734851 20633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:12.735456 20633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260924 23:10:12.735553 20633 leader_election.cc:302] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 748d3ede2acb4fdf92c751bdb2bf1bdf; no voters:
I20260924 23:10:12.735745 20633 leader_election.cc:288] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:12.735780 20638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:12.735898 20638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [term 1 LEADER]: Becoming Leader. State: Replica: 748d3ede2acb4fdf92c751bdb2bf1bdf, State: Running, Role: LEADER
I20260924 23:10:12.736011 20633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:12.735980 20638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } }
I20260924 23:10:12.736442 20640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748d3ede2acb4fdf92c751bdb2bf1bdf. Latest consensus state: current_term: 1 leader_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } } }
I20260924 23:10:12.737092 20640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:12.736407 20639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d3ede2acb4fdf92c751bdb2bf1bdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37907 } } }
I20260924 23:10:12.737174 20639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:12.737325 20651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:12.737787 20651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 23:10:12 dist-test-slave-ph9j krb5kdc[20552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:10:12.739624 20651 catalog_manager.cc:1347] Generated new cluster ID: cc6c6492ffcd4005a2ce40fdb8b77153
I20260924 23:10:12.739675 20651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:12.742863 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:12.731446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35097 (local address 127.0.0.1:37907)
0924 23:10:12.731618 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:12.731634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:12.731663 (+ 29us) server_negotiation.cc:407] Connection header received
0924 23:10:12.732139 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:12.732145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:12.732193 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:12.732303 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:12.733112 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:12.734079 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:12.735124 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:12.735550 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:12.735663 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:12.738998 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:12.739002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:12.739003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:12.739015 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:12.741135 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:12.741596 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:12.741599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:12.741599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:12.741698 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:12.741966 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:12.741967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:12.741967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:12.742275 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:12.742430 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:12.742573 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 23:10:12.742665 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260924 23:10:12.744099 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:12.752307 20651 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:12.752820 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:12.745569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41647 (local address 127.0.0.1:37907)
0924 23:10:12.745604 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:12.745605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:12.745764 (+ 159us) server_negotiation.cc:407] Connection header received
0924 23:10:12.745790 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:12.745791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:12.745800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:12.745864 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:12.746444 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:12.747202 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:12.748175 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:12.748512 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:12.748587 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:12.749911 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:12.749912 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:12.749913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:12.749920 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:12.751449 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:12.751870 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:12.751871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:12.751871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:12.751927 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:12.752188 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:12.752189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:12.752190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:12.752338 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:12.752439 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:12.752646 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 23:10:12.752717 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:12.753005 20651 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:12.767583 20651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 748d3ede2acb4fdf92c751bdb2bf1bdf: Generated new TSK 0
I20260924 23:10:12.767803 20651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:12.775713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20565
W20260924 23:10:12.780172 20550 connection.cc:535] client connection to 127.0.0.1:37907 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37907): Connection reset by peer (error 104)
2026-09-24T23:10:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:12 dist-test-slave-ph9j krb5kdc[20664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:12 dist-test-slave-ph9j krb5kdc[20664](info): set up 2 sockets
Sep 24 23:10:12 dist-test-slave-ph9j krb5kdc[20664](info): commencing operation
krb5kdc: starting...
W20260924 23:10:13.801961 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:13 dist-test-slave-ph9j krb5kdc[20664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:13.932914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45719
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:14.015695 20677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:14.015934 20677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:14.015983 20677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:14.017129 20677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:14.017200 20677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:14.017225 20677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:14.017244 20677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:14.017273 20677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:14.018759 20677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:14.019037 20677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:14.019273 20677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:14.024184 20683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:14.024194 20682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:14.024194 20685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:14.024474 20677 server_base.cc:1034] running on GCE node
I20260924 23:10:14.024706 20677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:14.024997 20677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:14.026166 20677 hybrid_clock.cc:648] HybridClock initialized: now 1790291414026152 us; error 46 us; skew 500 ppm
Sep 24 23:10:14 dist-test-slave-ph9j krb5kdc[20664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:14.028743 20677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:14.029189 20677 webserver.cc:460] Webserver started at http://127.0.0.1:39265/ using document root <none> and password file <none>
I20260924 23:10:14.029430 20677 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:14.029500 20677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:14.029655 20677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:14.030761 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "e7489fea0c0f4249bb3081c02c8c1a14"
format_stamp: "Formatted at 2026-09-24 23:10:14 on dist-test-slave-ph9j"
server_key: "76149de6c0fe8694a5a8b40abaa8482a"
server_key_iv: "aa500bbcf45044d0db187688c183d48a"
server_key_version: "encryptionkey@0"
I20260924 23:10:14.031131 20677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7489fea0c0f4249bb3081c02c8c1a14"
format_stamp: "Formatted at 2026-09-24 23:10:14 on dist-test-slave-ph9j"
server_key: "76149de6c0fe8694a5a8b40abaa8482a"
server_key_iv: "aa500bbcf45044d0db187688c183d48a"
server_key_version: "encryptionkey@0"
I20260924 23:10:14.032431 20677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:10:14.033154 20692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:14.033327 20677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:14.033433 20677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "e7489fea0c0f4249bb3081c02c8c1a14"
format_stamp: "Formatted at 2026-09-24 23:10:14 on dist-test-slave-ph9j"
server_key: "76149de6c0fe8694a5a8b40abaa8482a"
server_key_iv: "aa500bbcf45044d0db187688c183d48a"
server_key_version: "encryptionkey@0"
I20260924 23:10:14.033505 20677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:14.047024 20677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:14.049319 20677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:14.049484 20677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:14.053555 20677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45719
I20260924 23:10:14.053596 20744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45719 every 8 connection(s)
I20260924 23:10:14.054093 20677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:14.054971 20745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:14.057255 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14: Bootstrap starting.
I20260924 23:10:14.057533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20677
I20260924 23:10:14.057664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:14.057863 392 external_mini_cluster.cc:1428] Setting key 5c3eb7ccead4acbe8f829e2090826200
I20260924 23:10:14.057977 20745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:14.058311 20745 log.cc:824] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:14.059005 20745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14: No bootstrap required, opened a new log
I20260924 23:10:14.060818 20745 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260924 23:10:14.060976 20745 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:14.061035 20745 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7489fea0c0f4249bb3081c02c8c1a14, State: Initialized, Role: FOLLOWER
I20260924 23:10:14.061165 20745 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260924 23:10:14.061246 20745 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:14.061290 20745 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:14.061340 20745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:14.062033 20745 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260924 23:10:14.062141 20745 leader_election.cc:302] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7489fea0c0f4249bb3081c02c8c1a14; no voters:
I20260924 23:10:14.062330 20745 leader_election.cc:288] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:14.062381 20750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:14.062541 20750 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [term 1 LEADER]: Becoming Leader. State: Replica: e7489fea0c0f4249bb3081c02c8c1a14, State: Running, Role: LEADER
I20260924 23:10:14.062585 20745 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:14.062629 20750 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } }
I20260924 23:10:14.063688 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:14.058430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38819 (local address 127.0.0.1:45719)
0924 23:10:14.058604 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:14.058613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:14.058646 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:10:14.059169 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:14.059179 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:14.059245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:14.059394 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:14.059601 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:14.060492 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:14.061470 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:14.062128 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:14.062247 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:14.062990 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:14.062994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:14.062994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:14.063198 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:14.063238 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:14.063252 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:14.063357 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:14.063451 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 23:10:14.063482 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":103,"threads_started":1}
I20260924 23:10:14.064461 20751 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7489fea0c0f4249bb3081c02c8c1a14. Latest consensus state: current_term: 1 leader_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } } }
I20260924 23:10:14.064669 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:14.064702 20751 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:14.064477 20752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7489fea0c0f4249bb3081c02c8c1a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45719 } } }
I20260924 23:10:14.064862 20752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14 [sys.catalog]: This master's current role is: LEADER
W20260924 23:10:14.064997 20764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:14.065050 20764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:14.065193 20765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:14.065392 20765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:14.067144 20765 catalog_manager.cc:1347] Generated new cluster ID: d5b7f1451396476082c5e79745a98cb5
I20260924 23:10:14.067193 20765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:14.069531 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:14.066150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51977 (local address 127.0.0.1:45719)
0924 23:10:14.066189 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:14.066191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:14.066320 (+ 129us) server_negotiation.cc:407] Connection header received
0924 23:10:14.066367 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:14.066368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:14.066379 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:14.066439 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:14.066566 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:14.067241 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:14.068264 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:14.068568 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:14.068632 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:14.069296 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:14.069297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:14.069297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:14.069301 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:14.069314 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:14.069317 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:14.069370 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:14.069455 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 23:10:14.069464 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:10:14.074218 20765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:14.074705 20765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:14.084740 20765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7489fea0c0f4249bb3081c02c8c1a14: Generated new TSK 0
I20260924 23:10:14.084954 20765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:14.090318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20677
W20260924 23:10:14.094905 20662 connection.cc:535] client connection to 127.0.0.1:45719 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45719): Connection reset by peer (error 104)
2026-09-24T23:10:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:14 dist-test-slave-ph9j krb5kdc[20776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:14 dist-test-slave-ph9j krb5kdc[20776](info): set up 2 sockets
Sep 24 23:10:14 dist-test-slave-ph9j krb5kdc[20776](info): commencing operation
krb5kdc: starting...
W20260924 23:10:15.114073 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:15 dist-test-slave-ph9j krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:15.232895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:15.313716 20789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:15.313969 20789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:15.313994 20789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:15.315179 20789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:15.315220 20789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:15.315235 20789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:15.315248 20789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:15.315260 20789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:15.316722 20789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:15.316985 20789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:15.317232 20789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:15.322243 20795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:15.322271 20794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:15.322409 20789 server_base.cc:1034] running on GCE node
W20260924 23:10:15.322243 20797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:15.322731 20789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:15.323053 20789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:15.324205 20789 hybrid_clock.cc:648] HybridClock initialized: now 1790291415324184 us; error 45 us; skew 500 ppm
Sep 24 23:10:15 dist-test-slave-ph9j krb5kdc[20776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:15.330822 20789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:15.331349 20789 webserver.cc:460] Webserver started at http://127.0.0.1:40195/ using document root <none> and password file <none>
I20260924 23:10:15.331585 20789 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:15.331655 20789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:15.331820 20789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:15.332935 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "91130b7ecaab4ba4af01e090e59ee67f"
format_stamp: "Formatted at 2026-09-24 23:10:15 on dist-test-slave-ph9j"
server_key: "10b35645c29fbe218a92786917e8b608"
server_key_iv: "3dd8c9eb34c95dccc8b5bbba3e68af45"
server_key_version: "encryptionkey@0"
I20260924 23:10:15.333319 20789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "91130b7ecaab4ba4af01e090e59ee67f"
format_stamp: "Formatted at 2026-09-24 23:10:15 on dist-test-slave-ph9j"
server_key: "10b35645c29fbe218a92786917e8b608"
server_key_iv: "3dd8c9eb34c95dccc8b5bbba3e68af45"
server_key_version: "encryptionkey@0"
I20260924 23:10:15.334779 20789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260924 23:10:15.335610 20804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:15.335811 20789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:15.335912 20789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "91130b7ecaab4ba4af01e090e59ee67f"
format_stamp: "Formatted at 2026-09-24 23:10:15 on dist-test-slave-ph9j"
server_key: "10b35645c29fbe218a92786917e8b608"
server_key_iv: "3dd8c9eb34c95dccc8b5bbba3e68af45"
server_key_version: "encryptionkey@0"
I20260924 23:10:15.335994 20789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:15.339522 20789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:15.342087 20789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:15.342247 20789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:15.346252 20789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41277
I20260924 23:10:15.346287 20856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41277 every 8 connection(s)
I20260924 23:10:15.346628 20789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:15.347612 20857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:15.347709 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20789
I20260924 23:10:15.347827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:15.348023 392 external_mini_cluster.cc:1428] Setting key 3a997c6fe8b5940ba0b852433dc29c22
I20260924 23:10:15.350478 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:15.348566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46321 (local address 127.0.0.1:41277)
0924 23:10:15.348722 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:15.348729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:15.348754 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:10:15.349195 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:15.349201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:15.349226 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:15.349339 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:15.350037 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:15.350040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:15.350041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:15.350044 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:15.350081 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:15.350091 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:15.350146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:15.350234 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:10:15.350285 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260924 23:10:15.351229 20857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f: Bootstrap starting.
I20260924 23:10:15.351456 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:15.351965 20857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:15.352368 20857 log.cc:824] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:15.353135 20857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f: No bootstrap required, opened a new log
I20260924 23:10:15.354143 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:15.353058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58621 (local address 127.0.0.1:41277)
0924 23:10:15.353098 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:15.353100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:15.353250 (+ 150us) server_negotiation.cc:407] Connection header received
0924 23:10:15.353277 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:15.353278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:15.353283 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:15.353346 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:15.353941 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:15.353943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:15.353943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:15.353947 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:15.353960 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:15.353965 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:15.354018 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:15.354064 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 23:10:15.354082 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:10:15.355326 20857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91130b7ecaab4ba4af01e090e59ee67f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41277 } }
I20260924 23:10:15.355456 20857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:15.355479 20857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91130b7ecaab4ba4af01e090e59ee67f, State: Initialized, Role: FOLLOWER
I20260924 23:10:15.355557 20857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91130b7ecaab4ba4af01e090e59ee67f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41277 } }
I20260924 23:10:15.355623 20857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:15.355644 20857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:15.355676 20857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:15.356145 20857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91130b7ecaab4ba4af01e090e59ee67f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41277 } }
I20260924 23:10:15.356227 20857 leader_election.cc:302] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91130b7ecaab4ba4af01e090e59ee67f; no voters:
I20260924 23:10:15.356356 20857 leader_election.cc:288] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:15.356433 20867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:15.356604 20857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:15.356630 20867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [term 1 LEADER]: Becoming Leader. State: Replica: 91130b7ecaab4ba4af01e090e59ee67f, State: Running, Role: LEADER
I20260924 23:10:15.356717 20867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91130b7ecaab4ba4af01e090e59ee67f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41277 } }
I20260924 23:10:15.357059 20868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91130b7ecaab4ba4af01e090e59ee67f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91130b7ecaab4ba4af01e090e59ee67f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41277 } } }
I20260924 23:10:15.357147 20868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:15.357366 20869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91130b7ecaab4ba4af01e090e59ee67f. Latest consensus state: current_term: 1 leader_uuid: "91130b7ecaab4ba4af01e090e59ee67f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91130b7ecaab4ba4af01e090e59ee67f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41277 } } }
I20260924 23:10:15.357474 20869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:15.357625 20875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:15.358127 20875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:15.359735 20875 catalog_manager.cc:1347] Generated new cluster ID: b4deef54d3d24384aa96d38db6e57b35
I20260924 23:10:15.359773 20875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:15.370676 20875 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:15.370889 20875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:15.380797 20875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91130b7ecaab4ba4af01e090e59ee67f: Generated new TSK 0
I20260924 23:10:15.381039 20875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:15.387748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20789
2026-09-24T23:10:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1296 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:15 dist-test-slave-ph9j krb5kdc[20888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:15 dist-test-slave-ph9j krb5kdc[20888](info): set up 2 sockets
Sep 24 23:10:15 dist-test-slave-ph9j krb5kdc[20888](info): commencing operation
krb5kdc: starting...
W20260924 23:10:16.489987 392 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:16 dist-test-slave-ph9j krb5kdc[20888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:16.612926 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:16.696022 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:16.696278 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:16.696321 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:16.697499 20901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:16.697575 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:16.697599 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:16.697628 20901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:16.697659 20901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:16.699158 20901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:16.699447 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:16.699682 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:16.704880 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:16.704937 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:16.704955 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:16.705082 20901 server_base.cc:1034] running on GCE node
I20260924 23:10:16.705431 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:16.705741 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:16.706909 20901 hybrid_clock.cc:648] HybridClock initialized: now 1790291416706911 us; error 42 us; skew 500 ppm
Sep 24 23:10:16 dist-test-slave-ph9j krb5kdc[20888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:16.709524 20901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:16.710016 20901 webserver.cc:460] Webserver started at http://127.0.0.1:40705/ using document root <none> and password file <none>
I20260924 23:10:16.710223 20901 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:16.710263 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:16.710352 20901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:16.711364 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "ca093d47b3a7452c8ce015bd880f556e"
format_stamp: "Formatted at 2026-09-24 23:10:16 on dist-test-slave-ph9j"
server_key: "bb2b6e987187e1ed787e0fb22d3e3ef9"
server_key_iv: "165f7d3bd03aa628661c6ee41cf97eae"
server_key_version: "encryptionkey@0"
I20260924 23:10:16.711663 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca093d47b3a7452c8ce015bd880f556e"
format_stamp: "Formatted at 2026-09-24 23:10:16 on dist-test-slave-ph9j"
server_key: "bb2b6e987187e1ed787e0fb22d3e3ef9"
server_key_iv: "165f7d3bd03aa628661c6ee41cf97eae"
server_key_version: "encryptionkey@0"
I20260924 23:10:16.712924 20901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:16.713634 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:16.713814 20901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:16.713954 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "ca093d47b3a7452c8ce015bd880f556e"
format_stamp: "Formatted at 2026-09-24 23:10:16 on dist-test-slave-ph9j"
server_key: "bb2b6e987187e1ed787e0fb22d3e3ef9"
server_key_iv: "165f7d3bd03aa628661c6ee41cf97eae"
server_key_version: "encryptionkey@0"
I20260924 23:10:16.714032 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:16.740945 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:16.743477 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:16.743606 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:16.747658 20901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35083
I20260924 23:10:16.747712 20968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35083 every 8 connection(s)
I20260924 23:10:16.748234 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:16.748927 20969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:16.751356 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e: Bootstrap starting.
I20260924 23:10:16.751960 20969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:16.752246 20969 log.cc:824] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:16.753015 20969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e: No bootstrap required, opened a new log
I20260924 23:10:16.754743 20969 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca093d47b3a7452c8ce015bd880f556e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35083 } }
I20260924 23:10:16.754912 20969 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:16.754990 20969 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca093d47b3a7452c8ce015bd880f556e, State: Initialized, Role: FOLLOWER
I20260924 23:10:16.755162 20969 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca093d47b3a7452c8ce015bd880f556e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35083 } }
I20260924 23:10:16.755244 20969 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:16.755287 20969 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:16.755336 20969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:16.756281 20969 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca093d47b3a7452c8ce015bd880f556e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35083 } }
I20260924 23:10:16.756402 20969 leader_election.cc:302] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca093d47b3a7452c8ce015bd880f556e; no voters:
I20260924 23:10:16.756593 20969 leader_election.cc:288] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:16.756630 20972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:16.756867 20972 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [term 1 LEADER]: Becoming Leader. State: Replica: ca093d47b3a7452c8ce015bd880f556e, State: Running, Role: LEADER
I20260924 23:10:16.756852 20969 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:16.757051 20972 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca093d47b3a7452c8ce015bd880f556e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35083 } }
I20260924 23:10:16.757458 20973 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca093d47b3a7452c8ce015bd880f556e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca093d47b3a7452c8ce015bd880f556e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35083 } } }
I20260924 23:10:16.757449 20974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca093d47b3a7452c8ce015bd880f556e. Latest consensus state: current_term: 1 leader_uuid: "ca093d47b3a7452c8ce015bd880f556e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca093d47b3a7452c8ce015bd880f556e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35083 } } }
I20260924 23:10:16.757541 20973 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:16.757552 20974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:16.757772 20981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:16.758419 20981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:16.758451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 20901
I20260924 23:10:16.758570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:16.758792 392 external_mini_cluster.cc:1428] Setting key 910144b25badcbc752542598071414d3
I20260924 23:10:16.760481 20981 catalog_manager.cc:1347] Generated new cluster ID: 8dc32a6cbdfd43fb86280fd3ea49dd19
I20260924 23:10:16.760546 20981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:10:16 dist-test-slave-ph9j krb5kdc[20888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:10:16.768151 20981 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:16.768898 20981 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:16.769603 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:16.759259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54045 (local address 127.0.0.1:35083)
0924 23:10:16.759685 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:16.759694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:16.759734 (+ 40us) server_negotiation.cc:407] Connection header received
0924 23:10:16.759849 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:16.759857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:16.759918 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:16.760067 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:16.760781 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:16.761773 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:16.762805 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:16.763269 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:16.763414 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:16.765833 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:16.765838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:16.765839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:16.765857 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:16.767820 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:16.768262 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:16.768266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:16.768266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:16.768363 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:16.768585 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:16.768586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:16.768586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:16.768879 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:16.769094 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:16.769169 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 23:10:16.769432 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":383,"thread_start_us":64,"threads_started":1}
I20260924 23:10:16.770583 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:16.776870 20981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca093d47b3a7452c8ce015bd880f556e: Generated new TSK 0
I20260924 23:10:16.777110 20981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:16.780587 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:16.771994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37365 (local address 127.0.0.1:35083)
0924 23:10:16.772032 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:16.772034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:16.774176 (+ 2142us) server_negotiation.cc:407] Connection header received
0924 23:10:16.774258 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:16.774259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:16.774271 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:16.774327 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:16.774924 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:16.775449 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:16.776262 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:16.776601 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:16.776685 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:16.777994 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:16.777995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:16.777996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:16.778003 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:16.779509 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:16.779856 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:16.779857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:16.779857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:16.779916 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:16.780104 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:16.780105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:16.780105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:16.780235 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:16.780337 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:16.780411 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 23:10:16.780493 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:16.782610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 20901
2026-09-24T23:10:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:16 dist-test-slave-ph9j krb5kdc[21000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:16 dist-test-slave-ph9j krb5kdc[21000](info): set up 2 sockets
Sep 24 23:10:16 dist-test-slave-ph9j krb5kdc[21000](info): commencing operation
krb5kdc: starting...
W20260924 23:10:17.830134 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:17 dist-test-slave-ph9j krb5kdc[21000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:17.949179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:18.032514 21013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:18.032754 21013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:18.032790 21013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:18.034014 21013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:18.034091 21013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:18.034116 21013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:18.034138 21013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:18.034166 21013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:18.035590 21013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:18.035846 21013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:18.036073 21013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:18.040894 21019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:18.040947 21018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:18.041144 21021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:18.041106 21013 server_base.cc:1034] running on GCE node
I20260924 23:10:18.041370 21013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:18.041664 21013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:18.042824 21013 hybrid_clock.cc:648] HybridClock initialized: now 1790291418042800 us; error 57 us; skew 500 ppm
Sep 24 23:10:18 dist-test-slave-ph9j krb5kdc[21000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:18.045269 21013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:18.045723 21013 webserver.cc:460] Webserver started at http://127.0.0.1:39389/ using document root <none> and password file <none>
I20260924 23:10:18.046017 21013 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:18.046087 21013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:18.046234 21013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:18.047246 21013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "c3897e660cb845cfb0e8763ce42076ed"
format_stamp: "Formatted at 2026-09-24 23:10:18 on dist-test-slave-ph9j"
server_key: "bfcb46451f0973aefd05f85287752c8f"
server_key_iv: "35a6c8bb93c2e5683e660dfee0d96dfe"
server_key_version: "encryptionkey@0"
I20260924 23:10:18.047600 21013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3897e660cb845cfb0e8763ce42076ed"
format_stamp: "Formatted at 2026-09-24 23:10:18 on dist-test-slave-ph9j"
server_key: "bfcb46451f0973aefd05f85287752c8f"
server_key_iv: "35a6c8bb93c2e5683e660dfee0d96dfe"
server_key_version: "encryptionkey@0"
I20260924 23:10:18.048992 21013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:10:18.049799 21028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:18.049990 21013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:18.050137 21013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "c3897e660cb845cfb0e8763ce42076ed"
format_stamp: "Formatted at 2026-09-24 23:10:18 on dist-test-slave-ph9j"
server_key: "bfcb46451f0973aefd05f85287752c8f"
server_key_iv: "35a6c8bb93c2e5683e660dfee0d96dfe"
server_key_version: "encryptionkey@0"
I20260924 23:10:18.050221 21013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:18.062755 21013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:18.065191 21013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:18.065372 21013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:18.069588 21013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34255
I20260924 23:10:18.069623 21080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34255 every 8 connection(s)
I20260924 23:10:18.070055 21013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:18.070937 21081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:18.073071 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed: Bootstrap starting.
I20260924 23:10:18.073599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21013
I20260924 23:10:18.073679 21081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:18.073717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:18.073956 392 external_mini_cluster.cc:1428] Setting key 95e16c6f35235984d72fd278ad5f06a5
I20260924 23:10:18.074071 21081 log.cc:824] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:18.074872 21081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed: No bootstrap required, opened a new log
I20260924 23:10:18.076998 21081 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3897e660cb845cfb0e8763ce42076ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34255 } }
I20260924 23:10:18.077158 21081 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:18.077189 21081 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3897e660cb845cfb0e8763ce42076ed, State: Initialized, Role: FOLLOWER
I20260924 23:10:18.077297 21081 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3897e660cb845cfb0e8763ce42076ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34255 } }
I20260924 23:10:18.077361 21081 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:18.077389 21081 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:18.077435 21081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:18.078112 21081 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3897e660cb845cfb0e8763ce42076ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34255 } }
I20260924 23:10:18.078220 21081 leader_election.cc:302] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3897e660cb845cfb0e8763ce42076ed; no voters:
I20260924 23:10:18.078389 21081 leader_election.cc:288] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:18.078449 21086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:18.078648 21086 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [term 1 LEADER]: Becoming Leader. State: Replica: c3897e660cb845cfb0e8763ce42076ed, State: Running, Role: LEADER
I20260924 23:10:18.078645 21081 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:18.078828 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3897e660cb845cfb0e8763ce42076ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34255 } }
I20260924 23:10:18.079295 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3897e660cb845cfb0e8763ce42076ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3897e660cb845cfb0e8763ce42076ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34255 } } }
I20260924 23:10:18.079372 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:18.079768 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3897e660cb845cfb0e8763ce42076ed. Latest consensus state: current_term: 1 leader_uuid: "c3897e660cb845cfb0e8763ce42076ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3897e660cb845cfb0e8763ce42076ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34255 } } }
I20260924 23:10:18.079847 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:18.080044 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:18.074456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55633 (local address 127.0.0.1:34255)
0924 23:10:18.074609 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:18.074619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:18.074653 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:10:18.075126 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:18.075133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:18.075178 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:18.075287 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:18.075537 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:18.076405 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:18.077561 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:18.078212 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:18.078323 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:18.079082 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:18.079085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:18.079085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:18.079300 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:18.079341 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:18.079358 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:18.079433 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:18.079757 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 23:10:18.079822 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260924 23:10:18.080469 21092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:18.080889 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:18.081043 21092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:18.082803 21092 catalog_manager.cc:1347] Generated new cluster ID: 7015df315c764d599e89ab6d8d4a49eb
I20260924 23:10:18.082886 21092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:18.085605 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:18.082275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50969 (local address 127.0.0.1:34255)
0924 23:10:18.082319 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:18.082321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:18.082421 (+ 100us) server_negotiation.cc:407] Connection header received
0924 23:10:18.082477 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:18.082477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:18.082485 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:18.082556 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:18.082713 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:18.083408 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:18.084372 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:18.084652 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:18.084685 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:18.085390 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:18.085391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:18.085391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:18.085395 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:18.085408 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:18.085412 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:18.085435 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:18.085520 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 23:10:18.085538 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:18.116484 21092 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:18.117075 21092 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:18.120774 21092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3897e660cb845cfb0e8763ce42076ed: Generated new TSK 0
I20260924 23:10:18.120965 21092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:18.130455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21013
2026-09-24T23:10:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:18 dist-test-slave-ph9j krb5kdc[21112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:18 dist-test-slave-ph9j krb5kdc[21112](info): set up 2 sockets
Sep 24 23:10:18 dist-test-slave-ph9j krb5kdc[21112](info): commencing operation
krb5kdc: starting...
W20260924 23:10:19.181991 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:19 dist-test-slave-ph9j krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:19.301079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:19.381529 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:19.381728 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:19.381749 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:19.382912 21125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:19.382956 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:19.382970 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:19.382982 21125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:19.382994 21125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:19.384457 21125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:19.384702 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:19.384918 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:10:19.389813 21125 server_base.cc:1034] running on GCE node
W20260924 23:10:19.389787 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:19.389796 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:19.389807 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:19.390321 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:19.390686 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:19.391970 21125 hybrid_clock.cc:648] HybridClock initialized: now 1790291419391939 us; error 36 us; skew 500 ppm
Sep 24 23:10:19 dist-test-slave-ph9j krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:19.394495 21125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:19.394976 21125 webserver.cc:460] Webserver started at http://127.0.0.1:45181/ using document root <none> and password file <none>
I20260924 23:10:19.395212 21125 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:19.395282 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:19.395434 21125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:19.396502 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "f0a5091cbb1c4c0a9997e20217b42c56"
format_stamp: "Formatted at 2026-09-24 23:10:19 on dist-test-slave-ph9j"
server_key: "0b824a4a2d6df5f18e1baa1b03d0dec1"
server_key_iv: "19ff67a6bb39b52f470d2701706aa527"
server_key_version: "encryptionkey@0"
I20260924 23:10:19.396878 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0a5091cbb1c4c0a9997e20217b42c56"
format_stamp: "Formatted at 2026-09-24 23:10:19 on dist-test-slave-ph9j"
server_key: "0b824a4a2d6df5f18e1baa1b03d0dec1"
server_key_iv: "19ff67a6bb39b52f470d2701706aa527"
server_key_version: "encryptionkey@0"
I20260924 23:10:19.398288 21125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:19.399020 21140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:19.399196 21125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:19.399308 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "f0a5091cbb1c4c0a9997e20217b42c56"
format_stamp: "Formatted at 2026-09-24 23:10:19 on dist-test-slave-ph9j"
server_key: "0b824a4a2d6df5f18e1baa1b03d0dec1"
server_key_iv: "19ff67a6bb39b52f470d2701706aa527"
server_key_version: "encryptionkey@0"
I20260924 23:10:19.399381 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:19.402747 21125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:19.405033 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:19.405194 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:19.409224 21125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41707
I20260924 23:10:19.409269 21192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41707 every 8 connection(s)
I20260924 23:10:19.409622 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:19.410661 21193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:19.412927 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56: Bootstrap starting.
I20260924 23:10:19.413465 21193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:19.413699 21193 log.cc:824] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:19.414433 21193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56: No bootstrap required, opened a new log
I20260924 23:10:19.415537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21125
I20260924 23:10:19.415661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:19.415863 392 external_mini_cluster.cc:1428] Setting key 21a860600747dfdba431803129faf4eb
I20260924 23:10:19.415997 21193 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41707 } }
I20260924 23:10:19.416214 21193 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:19.416257 21193 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a5091cbb1c4c0a9997e20217b42c56, State: Initialized, Role: FOLLOWER
I20260924 23:10:19.416472 21193 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41707 } }
I20260924 23:10:19.416602 21193 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:19.416651 21193 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:19.416700 21193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:19.417834 21193 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41707 } }
I20260924 23:10:19.417975 21193 leader_election.cc:302] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0a5091cbb1c4c0a9997e20217b42c56; no voters:
I20260924 23:10:19.418165 21193 leader_election.cc:288] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:19.418229 21198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:19.418257 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:19.416462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47675 (local address 127.0.0.1:41707)
0924 23:10:19.416723 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:19.416733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:19.416766 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:10:19.416892 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:19.416899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:19.416928 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:19.417098 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:19.417737 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:19.417741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:19.417741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:19.417745 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:19.417791 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:19.417806 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:19.417899 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:19.418001 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 23:10:19.418053 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260924 23:10:19.418421 21193 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:19.418406 21198 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [term 1 LEADER]: Becoming Leader. State: Replica: f0a5091cbb1c4c0a9997e20217b42c56, State: Running, Role: LEADER
I20260924 23:10:19.418678 21198 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41707 } }
I20260924 23:10:19.419094 21199 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41707 } } }
I20260924 23:10:19.419236 21199 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:19.419348 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:19.419744 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:19.420065 21200 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0a5091cbb1c4c0a9997e20217b42c56. Latest consensus state: current_term: 1 leader_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0a5091cbb1c4c0a9997e20217b42c56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41707 } } }
I20260924 23:10:19.420171 21200 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:19.420302 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:19.422068 21207 catalog_manager.cc:1347] Generated new cluster ID: 1da3cd6fee394497a4b1c1f8be0a2f21
I20260924 23:10:19.422124 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:19.422442 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:19.421173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38081 (local address 127.0.0.1:41707)
0924 23:10:19.421206 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:19.421208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:19.421395 (+ 187us) server_negotiation.cc:407] Connection header received
0924 23:10:19.421425 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:19.421425 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:19.421431 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:19.421495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:19.422188 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:19.422190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:19.422190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:19.422194 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:19.422206 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:19.422212 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:19.422286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:19.422343 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 23:10:19.422358 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:10:19.438717 21207 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:19.438918 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:19.443503 21207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0a5091cbb1c4c0a9997e20217b42c56: Generated new TSK 0
I20260924 23:10:19.443746 21207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:19.452252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21125
2026-09-24T23:10:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:19 dist-test-slave-ph9j krb5kdc[21224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:19 dist-test-slave-ph9j krb5kdc[21224](info): set up 2 sockets
Sep 24 23:10:19 dist-test-slave-ph9j krb5kdc[21224](info): commencing operation
krb5kdc: starting...
W20260924 23:10:20.481863 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:20 dist-test-slave-ph9j krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:20.600217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:20.680667 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:20.680904 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:20.680943 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:20.682134 21237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:20.682197 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:20.682224 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:20.682255 21237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:20.682286 21237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:20.683735 21237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:20.683992 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:20.684250 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:20.689064 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:20.689075 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:20.689153 21237 server_base.cc:1034] running on GCE node
W20260924 23:10:20.689075 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:20.689464 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:20.689713 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:20.690852 21237 hybrid_clock.cc:648] HybridClock initialized: now 1790291420690836 us; error 31 us; skew 500 ppm
Sep 24 23:10:20 dist-test-slave-ph9j krb5kdc[21224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:20.700851 21237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:20.701300 21237 webserver.cc:460] Webserver started at http://127.0.0.1:43769/ using document root <none> and password file <none>
I20260924 23:10:20.701524 21237 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:20.701570 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:20.701745 21237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:20.702868 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "55f98fb01ca947f194eff036a83c7dea"
format_stamp: "Formatted at 2026-09-24 23:10:20 on dist-test-slave-ph9j"
server_key: "6bfc0c1539073a279a290c092da3d677"
server_key_iv: "80d439d15dd6aa068eb3a75a56d83b27"
server_key_version: "encryptionkey@0"
I20260924 23:10:20.703236 21237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "55f98fb01ca947f194eff036a83c7dea"
format_stamp: "Formatted at 2026-09-24 23:10:20 on dist-test-slave-ph9j"
server_key: "6bfc0c1539073a279a290c092da3d677"
server_key_iv: "80d439d15dd6aa068eb3a75a56d83b27"
server_key_version: "encryptionkey@0"
I20260924 23:10:20.704545 21237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:20.705262 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:20.705433 21237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:20.705519 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "55f98fb01ca947f194eff036a83c7dea"
format_stamp: "Formatted at 2026-09-24 23:10:20 on dist-test-slave-ph9j"
server_key: "6bfc0c1539073a279a290c092da3d677"
server_key_iv: "80d439d15dd6aa068eb3a75a56d83b27"
server_key_version: "encryptionkey@0"
I20260924 23:10:20.705605 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:20.726904 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:20.729276 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:20.729444 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:20.733798 21237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42487
I20260924 23:10:20.733834 21304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42487 every 8 connection(s)
I20260924 23:10:20.734320 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:20.734937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21237
I20260924 23:10:20.735051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:20.735284 392 external_mini_cluster.cc:1428] Setting key 41d6263f132d100db00326230789fc5d
I20260924 23:10:20.735451 21305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:20.738826 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea: Bootstrap starting.
I20260924 23:10:20.739542 21305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:20.739850 21305 log.cc:824] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:20.740602 21305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea: No bootstrap required, opened a new log
Sep 24 23:10:20 dist-test-slave-ph9j krb5kdc[21224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:10:20.743090 21305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f98fb01ca947f194eff036a83c7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260924 23:10:20.743230 21305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:20.743253 21305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f98fb01ca947f194eff036a83c7dea, State: Initialized, Role: FOLLOWER
I20260924 23:10:20.743404 21305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f98fb01ca947f194eff036a83c7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260924 23:10:20.743466 21305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:20.743487 21305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:20.743561 21305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:20.744196 21305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f98fb01ca947f194eff036a83c7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260924 23:10:20.744303 21305 leader_election.cc:302] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55f98fb01ca947f194eff036a83c7dea; no voters:
I20260924 23:10:20.744482 21305 leader_election.cc:288] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:20.744513 21310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:20.744627 21310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [term 1 LEADER]: Becoming Leader. State: Replica: 55f98fb01ca947f194eff036a83c7dea, State: Running, Role: LEADER
I20260924 23:10:20.744753 21305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:20.744771 21310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f98fb01ca947f194eff036a83c7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } }
I20260924 23:10:20.745186 21310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55f98fb01ca947f194eff036a83c7dea. Latest consensus state: current_term: 1 leader_uuid: "55f98fb01ca947f194eff036a83c7dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f98fb01ca947f194eff036a83c7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } } }
I20260924 23:10:20.745455 21310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:20.745366 21311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55f98fb01ca947f194eff036a83c7dea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55f98fb01ca947f194eff036a83c7dea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42487 } } }
I20260924 23:10:20.746033 21311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:20.746065 21319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:20.746757 21319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:20.747268 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:20.736027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:56537 (local address 127.0.0.1:42487)
0924 23:10:20.736245 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:20.736254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:20.736287 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:10:20.736726 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:20.736735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:20.736805 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:20.736952 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:20.737925 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:20.738816 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:20.739901 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:20.740462 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:20.740543 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:20.743056 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:20.743060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:20.743061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:20.743071 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:20.745127 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:20.745635 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:20.745638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:20.745639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:20.745721 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:20.746017 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:20.746018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:20.746018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:20.746388 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:20.746899 (+ 511us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:20.746983 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:10:20.747079 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260924 23:10:20.748504 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:20.748556 21319 catalog_manager.cc:1347] Generated new cluster ID: c86aec27078243b08c9b6c087f2916de
I20260924 23:10:20.748600 21319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:20.755957 21308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:20.749755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:44251 (local address 127.0.0.1:42487)
0924 23:10:20.749792 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:20.749794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:20.749922 (+ 128us) server_negotiation.cc:407] Connection header received
0924 23:10:20.750013 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:20.750014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:20.750023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:20.750067 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:20.750602 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:20.751069 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:20.751844 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:20.752075 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:20.752159 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:20.753328 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:20.753329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:20.753330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:20.753336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:20.754809 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:20.755173 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:20.755174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:20.755174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:20.755225 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:20.755447 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:20.755448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:20.755448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:20.755558 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:20.755672 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:20.755787 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 23:10:20.755848 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:20.801622 21319 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:20.802313 21319 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:20.813757 21319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55f98fb01ca947f194eff036a83c7dea: Generated new TSK 0
I20260924 23:10:20.814002 21319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:20.825116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21237
W20260924 23:10:20.829720 21222 connection.cc:535] client connection to 127.0.0.1:42487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42487): Connection reset by peer (error 104)
2026-09-24T23:10:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:20 dist-test-slave-ph9j krb5kdc[21336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:20 dist-test-slave-ph9j krb5kdc[21336](info): set up 2 sockets
Sep 24 23:10:20 dist-test-slave-ph9j krb5kdc[21336](info): commencing operation
krb5kdc: starting...
W20260924 23:10:21.858049 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:21 dist-test-slave-ph9j krb5kdc[21336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:21.976353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:22.055163 21349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:22.055394 21349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:22.055436 21349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:22.056573 21349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:22.056644 21349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:22.056669 21349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:22.056706 21349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:22.056756 21349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:22.058256 21349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:22.058533 21349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:22.058763 21349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:22.063638 21355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:22.063663 21354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:22.063689 21357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:22.063915 21349 server_base.cc:1034] running on GCE node
I20260924 23:10:22.064226 21349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:22.064527 21349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:22.065713 21349 hybrid_clock.cc:648] HybridClock initialized: now 1790291422065706 us; error 37 us; skew 500 ppm
Sep 24 23:10:22 dist-test-slave-ph9j krb5kdc[21336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:22.068356 21349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:22.068816 21349 webserver.cc:460] Webserver started at http://127.0.0.1:35179/ using document root <none> and password file <none>
I20260924 23:10:22.069072 21349 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:22.069144 21349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:22.069293 21349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:22.070452 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "ecf4412434db43d9bd4bcec7577c62f2"
format_stamp: "Formatted at 2026-09-24 23:10:22 on dist-test-slave-ph9j"
server_key: "822746826946acd4c6376eaea9ef1749"
server_key_iv: "28c077ce8afba1b5abad1f8b7a6420d4"
server_key_version: "encryptionkey@0"
I20260924 23:10:22.070816 21349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecf4412434db43d9bd4bcec7577c62f2"
format_stamp: "Formatted at 2026-09-24 23:10:22 on dist-test-slave-ph9j"
server_key: "822746826946acd4c6376eaea9ef1749"
server_key_iv: "28c077ce8afba1b5abad1f8b7a6420d4"
server_key_version: "encryptionkey@0"
I20260924 23:10:22.072096 21349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:22.072774 21364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:22.072948 21349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:22.073057 21349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "ecf4412434db43d9bd4bcec7577c62f2"
format_stamp: "Formatted at 2026-09-24 23:10:22 on dist-test-slave-ph9j"
server_key: "822746826946acd4c6376eaea9ef1749"
server_key_iv: "28c077ce8afba1b5abad1f8b7a6420d4"
server_key_version: "encryptionkey@0"
I20260924 23:10:22.073132 21349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:22.091040 21349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:22.093534 21349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:22.093707 21349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:22.097730 21349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33131
I20260924 23:10:22.097767 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33131 every 8 connection(s)
I20260924 23:10:22.098193 21349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:22.099086 21417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:22.101158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21349
I20260924 23:10:22.101308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:22.101477 21417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2: Bootstrap starting.
I20260924 23:10:22.101513 392 external_mini_cluster.cc:1428] Setting key a80d6ca8436c86feec1d448483c53d63
I20260924 23:10:22.102232 21417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:22.102636 21417 log.cc:824] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:22.103400 21417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2: No bootstrap required, opened a new log
I20260924 23:10:22.105641 21417 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf4412434db43d9bd4bcec7577c62f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260924 23:10:22.105826 21417 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:22.105863 21417 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf4412434db43d9bd4bcec7577c62f2, State: Initialized, Role: FOLLOWER
I20260924 23:10:22.106002 21417 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf4412434db43d9bd4bcec7577c62f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260924 23:10:22.106076 21417 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:22.106106 21417 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:22.106153 21417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:22.106822 21417 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf4412434db43d9bd4bcec7577c62f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260924 23:10:22.106941 21417 leader_election.cc:302] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf4412434db43d9bd4bcec7577c62f2; no voters:
I20260924 23:10:22.107112 21417 leader_election.cc:288] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:22.107152 21422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:22.107342 21422 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [term 1 LEADER]: Becoming Leader. State: Replica: ecf4412434db43d9bd4bcec7577c62f2, State: Running, Role: LEADER
I20260924 23:10:22.107372 21417 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:22.107493 21422 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf4412434db43d9bd4bcec7577c62f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260924 23:10:22.107913 21424 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf4412434db43d9bd4bcec7577c62f2. Latest consensus state: current_term: 1 leader_uuid: "ecf4412434db43d9bd4bcec7577c62f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf4412434db43d9bd4bcec7577c62f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } } }
I20260924 23:10:22.107892 21423 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf4412434db43d9bd4bcec7577c62f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf4412434db43d9bd4bcec7577c62f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } } }
I20260924 23:10:22.108081 21423 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:22.108256 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:22.102156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:35339 (local address 127.0.0.1:33131)
0924 23:10:22.102376 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:22.102385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:22.102422 (+ 37us) server_negotiation.cc:407] Connection header received
0924 23:10:22.103010 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:22.103017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:22.103107 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:22.103259 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:22.103493 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:22.104861 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:22.105732 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:22.106365 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:22.106484 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:22.107165 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:22.107170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:22.107170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:22.107387 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:22.107426 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:22.107443 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:22.107533 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:22.107809 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 23:10:22.107838 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260924 23:10:22.108709 21424 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:22.108868 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:22.109349 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:22.109396 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:22.111066 21429 catalog_manager.cc:1347] Generated new cluster ID: cf04556422fe44f0ad61795a112db1da
I20260924 23:10:22.111131 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:22.114711 21429 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:22.114712 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:22.110893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:53783 (local address 127.0.0.1:33131)
0924 23:10:22.110931 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:22.110933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:22.111181 (+ 248us) server_negotiation.cc:407] Connection header received
0924 23:10:22.111279 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:22.111280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:22.111292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:22.111350 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:22.111494 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:22.112241 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:22.113263 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:22.113574 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:22.113639 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:22.114422 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:22.114423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:22.114424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:22.114428 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:22.114444 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:22.114449 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:22.114521 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:22.114625 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 23:10:22.114635 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:22.115514 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:22.122776 21429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecf4412434db43d9bd4bcec7577c62f2: Generated new TSK 0
I20260924 23:10:22.123005 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:22.128402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21349
W20260924 23:10:22.132720 21334 connection.cc:535] client connection to 127.0.0.1:33131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33131): Connection reset by peer (error 104)
2026-09-24T23:10:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:22 dist-test-slave-ph9j krb5kdc[21448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:22 dist-test-slave-ph9j krb5kdc[21448](info): set up 2 sockets
Sep 24 23:10:22 dist-test-slave-ph9j krb5kdc[21448](info): commencing operation
krb5kdc: starting...
W20260924 23:10:23.161962 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:23 dist-test-slave-ph9j krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:23.280228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:23.359901 21461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:23.360154 21461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:23.360191 21461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:23.361317 21461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:23.361392 21461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:23.361418 21461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:23.361444 21461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:23.361475 21461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:23.362950 21461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:23.363229 21461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:23.363446 21461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:23.368307 21467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:23.368341 21469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:23.368451 21466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:23.368499 21461 server_base.cc:1034] running on GCE node
I20260924 23:10:23.368724 21461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:23.368993 21461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:23.370206 21461 hybrid_clock.cc:648] HybridClock initialized: now 1790291423370191 us; error 25 us; skew 500 ppm
Sep 24 23:10:23 dist-test-slave-ph9j krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:23.372748 21461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:23.373198 21461 webserver.cc:460] Webserver started at http://127.0.0.1:37953/ using document root <none> and password file <none>
I20260924 23:10:23.373430 21461 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:23.373505 21461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:23.373660 21461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:23.374866 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4"
format_stamp: "Formatted at 2026-09-24 23:10:23 on dist-test-slave-ph9j"
server_key: "54403f4e0cf9b14e7cb3b476d2e52295"
server_key_iv: "352d568b5e563cd29d42bc66e9be878d"
server_key_version: "encryptionkey@0"
I20260924 23:10:23.375231 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4"
format_stamp: "Formatted at 2026-09-24 23:10:23 on dist-test-slave-ph9j"
server_key: "54403f4e0cf9b14e7cb3b476d2e52295"
server_key_iv: "352d568b5e563cd29d42bc66e9be878d"
server_key_version: "encryptionkey@0"
I20260924 23:10:23.376556 21461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:23.377261 21476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:23.377431 21461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:23.377538 21461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4"
format_stamp: "Formatted at 2026-09-24 23:10:23 on dist-test-slave-ph9j"
server_key: "54403f4e0cf9b14e7cb3b476d2e52295"
server_key_iv: "352d568b5e563cd29d42bc66e9be878d"
server_key_version: "encryptionkey@0"
I20260924 23:10:23.377617 21461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:23.380854 21461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:23.383106 21461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:23.383266 21461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:23.387166 21461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38837
I20260924 23:10:23.387200 21528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38837 every 8 connection(s)
I20260924 23:10:23.387528 21461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:23.388621 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:23.391180 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4: Bootstrap starting.
I20260924 23:10:23.391765 21529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:23.392026 21529 log.cc:824] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:23.392779 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4: No bootstrap required, opened a new log
I20260924 23:10:23.394516 21529 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38837 } }
I20260924 23:10:23.394657 21529 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:23.394680 21529 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcde8e1e4e91425f8d41cd48ec9be9b4, State: Initialized, Role: FOLLOWER
I20260924 23:10:23.394817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21461
I20260924 23:10:23.394835 21529 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38837 } }
I20260924 23:10:23.394919 21529 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:23.394973 21529 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:23.395047 21529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:23.395052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:23.395301 392 external_mini_cluster.cc:1428] Setting key 7e6a156426d39b6456999e5cf8cf08bf
I20260924 23:10:23.395845 21529 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38837 } }
I20260924 23:10:23.395956 21529 leader_election.cc:302] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcde8e1e4e91425f8d41cd48ec9be9b4; no voters:
I20260924 23:10:23.396155 21529 leader_election.cc:288] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:23.396185 21534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:23.396296 21534 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [term 1 LEADER]: Becoming Leader. State: Replica: fcde8e1e4e91425f8d41cd48ec9be9b4, State: Running, Role: LEADER
I20260924 23:10:23.396433 21529 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:23.396432 21534 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38837 } }
I20260924 23:10:23.396946 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38837 } } }
I20260924 23:10:23.397161 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:23.397104 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcde8e1e4e91425f8d41cd48ec9be9b4. Latest consensus state: current_term: 1 leader_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcde8e1e4e91425f8d41cd48ec9be9b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38837 } } }
I20260924 23:10:23.397420 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:23.397614 21540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:23.398118 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:23.395856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:35119 (local address 127.0.0.1:38837)
0924 23:10:23.396223 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:23.396233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:23.396276 (+ 43us) server_negotiation.cc:407] Connection header received
0924 23:10:23.396395 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:23.396403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:23.396439 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:23.396886 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:23.397351 (+ 465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:23.397355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:23.397355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:23.397359 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:23.397399 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:23.397414 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:23.397701 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:23.397736 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 23:10:23.397795 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":84,"threads_started":1}
I20260924 23:10:23.398527 21540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:23.399184 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:23.401202 21540 catalog_manager.cc:1347] Generated new cluster ID: 59302ab5ff81450eb0882c64037fa736
I20260924 23:10:23.401258 21540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:23.402035 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:23.400726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:41465 (local address 127.0.0.1:38837)
0924 23:10:23.400768 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:23.400770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:23.401011 (+ 241us) server_negotiation.cc:407] Connection header received
0924 23:10:23.401070 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:23.401072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:23.401077 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:23.401135 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:23.401798 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:23.401799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:23.401800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:23.401804 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:23.401820 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:23.401825 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:23.401870 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:23.401953 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 23:10:23.401969 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:10:23.419947 21540 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:23.420176 21540 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:23.424695 21540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcde8e1e4e91425f8d41cd48ec9be9b4: Generated new TSK 0
I20260924 23:10:23.424907 21540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:23.432247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21461
2026-09-24T23:10:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:23 dist-test-slave-ph9j krb5kdc[21560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:23 dist-test-slave-ph9j krb5kdc[21560](info): set up 2 sockets
Sep 24 23:10:23 dist-test-slave-ph9j krb5kdc[21560](info): commencing operation
krb5kdc: starting...
W20260924 23:10:24.466070 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:24 dist-test-slave-ph9j krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:24.592412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:24.674567 21573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:24.674793 21573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:24.674831 21573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:24.675951 21573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:24.676018 21573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:24.676043 21573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:24.676071 21573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:24.676110 21573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:24.677520 21573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:24.677788 21573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:24.678016 21573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:24.682786 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:24.682792 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:24.683023 21573 server_base.cc:1034] running on GCE node
W20260924 23:10:24.683068 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:24.683291 21573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:24.683569 21573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:24.684753 21573 hybrid_clock.cc:648] HybridClock initialized: now 1790291424684738 us; error 29 us; skew 500 ppm
Sep 24 23:10:24 dist-test-slave-ph9j krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:24.687556 21573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:24.687985 21573 webserver.cc:460] Webserver started at http://127.0.0.1:36003/ using document root <none> and password file <none>
I20260924 23:10:24.688225 21573 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:24.688290 21573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:24.688439 21573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:24.689505 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "496f292188504dd5b7dd273e922658dd"
format_stamp: "Formatted at 2026-09-24 23:10:24 on dist-test-slave-ph9j"
server_key: "69a77b98b5b7bc89abe1d496a1bce6e4"
server_key_iv: "63b4906ef53374d2bd5690841759bd1d"
server_key_version: "encryptionkey@0"
I20260924 23:10:24.689864 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "496f292188504dd5b7dd273e922658dd"
format_stamp: "Formatted at 2026-09-24 23:10:24 on dist-test-slave-ph9j"
server_key: "69a77b98b5b7bc89abe1d496a1bce6e4"
server_key_iv: "63b4906ef53374d2bd5690841759bd1d"
server_key_version: "encryptionkey@0"
I20260924 23:10:24.691294 21573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:24.692185 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:24.692378 21573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:24.692478 21573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "496f292188504dd5b7dd273e922658dd"
format_stamp: "Formatted at 2026-09-24 23:10:24 on dist-test-slave-ph9j"
server_key: "69a77b98b5b7bc89abe1d496a1bce6e4"
server_key_iv: "63b4906ef53374d2bd5690841759bd1d"
server_key_version: "encryptionkey@0"
I20260924 23:10:24.692557 21573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:24.711558 21573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:24.713994 21573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:24.714154 21573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:24.717940 21573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46127
I20260924 23:10:24.717969 21640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46127 every 8 connection(s)
I20260924 23:10:24.718333 21573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:24.719182 21641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:24.721396 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd: Bootstrap starting.
I20260924 23:10:24.721999 21641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:24.722289 21641 log.cc:824] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:24.722954 21641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd: No bootstrap required, opened a new log
I20260924 23:10:24.724489 21641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496f292188504dd5b7dd273e922658dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260924 23:10:24.724642 21641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:24.724666 21641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 496f292188504dd5b7dd273e922658dd, State: Initialized, Role: FOLLOWER
I20260924 23:10:24.724851 21641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496f292188504dd5b7dd273e922658dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260924 23:10:24.724915 21641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:24.724936 21641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:24.725008 21641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:24.725528 21641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496f292188504dd5b7dd273e922658dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260924 23:10:24.725641 21641 leader_election.cc:302] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 496f292188504dd5b7dd273e922658dd; no voters:
I20260924 23:10:24.725814 21641 leader_election.cc:288] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:24.725872 21644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:24.726063 21644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [term 1 LEADER]: Becoming Leader. State: Replica: 496f292188504dd5b7dd273e922658dd, State: Running, Role: LEADER
I20260924 23:10:24.726095 21641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:24.726159 21644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496f292188504dd5b7dd273e922658dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } }
I20260924 23:10:24.726506 21645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "496f292188504dd5b7dd273e922658dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496f292188504dd5b7dd273e922658dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } } }
I20260924 23:10:24.726509 21646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 496f292188504dd5b7dd273e922658dd. Latest consensus state: current_term: 1 leader_uuid: "496f292188504dd5b7dd273e922658dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "496f292188504dd5b7dd273e922658dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46127 } } }
I20260924 23:10:24.726642 21645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:24.726706 21646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:24.726897 21648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:24.727402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21573
I20260924 23:10:24.727460 21648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:24.727512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:24.727721 392 external_mini_cluster.cc:1428] Setting key 438d51b29f9d96a381cbfebc8b96ccce
I20260924 23:10:24.729034 21648 catalog_manager.cc:1347] Generated new cluster ID: 57479775c76d4f988d7ae01056068218
I20260924 23:10:24.729089 21648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:10:24 dist-test-slave-ph9j krb5kdc[21560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 23:10:24.737402 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:24.728125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:59251 (local address 127.0.0.1:46127)
0924 23:10:24.728357 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:24.728367 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:24.728413 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:10:24.728519 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:24.728524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:24.728602 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:24.728741 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:24.729570 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:24.730589 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:24.731361 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:24.731778 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:24.731901 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:24.734270 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:24.734273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:24.734274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:24.734286 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:24.735897 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:24.736279 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:24.736282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:24.736282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:24.736380 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:24.736612 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:24.736613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:24.736613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:24.736839 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:24.737016 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:24.737097 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 23:10:24.737215 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260924 23:10:24.738374 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:24.746611 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:24.739936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:37537 (local address 127.0.0.1:46127)
0924 23:10:24.739976 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:24.739978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:24.739989 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:10:24.740102 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:24.740103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:24.740114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:24.740166 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:24.740713 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:24.741272 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:24.742148 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:24.742473 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:24.742510 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:24.743805 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:24.743807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:24.743807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:24.743815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:24.745300 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:24.745688 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:24.745689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:24.745689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:24.745747 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:24.745991 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:24.745992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:24.745992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:24.746108 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:24.746226 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:24.746359 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 23:10:24.746426 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:10:24.751103 21648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:24.751837 21648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:24.756064 21648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 496f292188504dd5b7dd273e922658dd: Generated new TSK 0
I20260924 23:10:24.756259 21648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:24.761279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21573
W20260924 23:10:24.765516 21558 connection.cc:535] client connection to 127.0.0.1:46127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46127): Connection reset by peer (error 104)
2026-09-24T23:10:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:24 dist-test-slave-ph9j krb5kdc[21672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:24 dist-test-slave-ph9j krb5kdc[21672](info): set up 2 sockets
Sep 24 23:10:24 dist-test-slave-ph9j krb5kdc[21672](info): commencing operation
krb5kdc: starting...
W20260924 23:10:25.797940 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:25 dist-test-slave-ph9j krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:25.916273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38091
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:25.995726 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:25.995965 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:25.996003 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:25.997125 21685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:25.997196 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:25.997221 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:25.997249 21685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:25.997280 21685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:25.998764 21685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:25.999024 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:25.999243 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:26.003984 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:26.003988 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:26.004251 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:26.004213 21685 server_base.cc:1034] running on GCE node
I20260924 23:10:26.004506 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:26.004791 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:26.005959 21685 hybrid_clock.cc:648] HybridClock initialized: now 1790291426005940 us; error 58 us; skew 500 ppm
Sep 24 23:10:26 dist-test-slave-ph9j krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:26.011734 21685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:26.012199 21685 webserver.cc:460] Webserver started at http://127.0.0.1:34593/ using document root <none> and password file <none>
I20260924 23:10:26.012428 21685 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:26.012501 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:26.012645 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:26.013733 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0"
format_stamp: "Formatted at 2026-09-24 23:10:26 on dist-test-slave-ph9j"
server_key: "4f47b9228a054344202496cb39a2b481"
server_key_iv: "465a6a5c88adb18035476a473c7a28dc"
server_key_version: "encryptionkey@0"
I20260924 23:10:26.014134 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0"
format_stamp: "Formatted at 2026-09-24 23:10:26 on dist-test-slave-ph9j"
server_key: "4f47b9228a054344202496cb39a2b481"
server_key_iv: "465a6a5c88adb18035476a473c7a28dc"
server_key_version: "encryptionkey@0"
I20260924 23:10:26.015592 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:26.016388 21700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:26.016615 21685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:26.016736 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0"
format_stamp: "Formatted at 2026-09-24 23:10:26 on dist-test-slave-ph9j"
server_key: "4f47b9228a054344202496cb39a2b481"
server_key_iv: "465a6a5c88adb18035476a473c7a28dc"
server_key_version: "encryptionkey@0"
I20260924 23:10:26.016821 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:26.033571 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:26.035852 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:26.036046 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:26.040192 21685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38091
I20260924 23:10:26.040232 21752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38091 every 8 connection(s)
I20260924 23:10:26.040592 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:26.040917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21685
I20260924 23:10:26.041040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:26.041253 392 external_mini_cluster.cc:1428] Setting key 656d9308a02f696e0a0ebce113889eab
I20260924 23:10:26.041757 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:26.045111 21753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0: Bootstrap starting.
I20260924 23:10:26.045785 21753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:26.046105 21753 log.cc:824] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:26.046744 21753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0: No bootstrap required, opened a new log
I20260924 23:10:26.047070 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:26.041871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:51563 (local address 127.0.0.1:38091)
0924 23:10:26.042065 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:26.042074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:26.042104 (+ 30us) server_negotiation.cc:407] Connection header received
0924 23:10:26.042653 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:26.042663 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:26.042729 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:26.042879 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:26.043078 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:26.044426 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:26.045225 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:26.045647 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:26.045770 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:26.046494 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:26.046498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:26.046498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:26.046664 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:26.046695 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:26.046705 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:26.046795 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:26.046874 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 23:10:26.046901 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260924 23:10:26.048015 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:26.049088 21753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260924 23:10:26.049299 21753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:26.049356 21753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d73bbda1c24f1ba40c0a6d663ed5e0, State: Initialized, Role: FOLLOWER
I20260924 23:10:26.049451 21753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260924 23:10:26.049510 21753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:26.049532 21753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:26.049559 21753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:26.050112 21753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260924 23:10:26.050194 21753 leader_election.cc:302] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95d73bbda1c24f1ba40c0a6d663ed5e0; no voters:
I20260924 23:10:26.050336 21753 leader_election.cc:288] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:26.050374 21763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:26.050494 21763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 95d73bbda1c24f1ba40c0a6d663ed5e0, State: Running, Role: LEADER
I20260924 23:10:26.050591 21753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:26.050655 21763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } }
I20260924 23:10:26.051009 21765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d73bbda1c24f1ba40c0a6d663ed5e0. Latest consensus state: current_term: 1 leader_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } } }
I20260924 23:10:26.051100 21765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:26.051280 21764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95d73bbda1c24f1ba40c0a6d663ed5e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38091 } } }
I20260924 23:10:26.051350 21770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:26.051440 21764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:26.051997 21770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:26.053542 21770 catalog_manager.cc:1347] Generated new cluster ID: 849f0c360f61465d954b521a3f4225d1
I20260924 23:10:26.053599 21770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:26.055249 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:26.049286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:39577 (local address 127.0.0.1:38091)
0924 23:10:26.049322 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:26.049324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:26.052122 (+ 2798us) server_negotiation.cc:407] Connection header received
0924 23:10:26.052235 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:26.052236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:26.052251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:26.052315 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:26.052484 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:26.053171 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:26.054083 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:26.054294 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:26.054372 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:26.055010 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:26.055011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:26.055012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:26.055016 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:26.055031 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:26.055035 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:26.055088 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:26.055149 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 23:10:26.055157 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:26.068459 21770 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:26.069170 21770 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:26.072877 21770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d73bbda1c24f1ba40c0a6d663ed5e0: Generated new TSK 0
I20260924 23:10:26.073228 21770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:26.075459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21685
W20260924 23:10:26.080399 21670 connection.cc:535] client connection to 127.0.0.1:38091 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38091): Connection reset by peer (error 104)
2026-09-24T23:10:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:26 dist-test-slave-ph9j krb5kdc[21784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:26 dist-test-slave-ph9j krb5kdc[21784](info): set up 2 sockets
Sep 24 23:10:26 dist-test-slave-ph9j krb5kdc[21784](info): commencing operation
krb5kdc: starting...
W20260924 23:10:27.110158 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:27 dist-test-slave-ph9j krb5kdc[21784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 23:10:27.228232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:27.308382 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:27.308620 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:27.308658 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:27.309810 21797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:27.309875 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:27.309921 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:27.309943 21797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:27.309970 21797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:27.311427 21797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:27.311702 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:27.311921 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:27.316756 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:27.316823 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:27.316833 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:27.316864 21797 server_base.cc:1034] running on GCE node
I20260924 23:10:27.317247 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:27.317498 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:27.318634 21797 hybrid_clock.cc:648] HybridClock initialized: now 1790291427318623 us; error 41 us; skew 500 ppm
Sep 24 23:10:27 dist-test-slave-ph9j krb5kdc[21784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:27.321107 21797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 23:10:27.321539 21797 webserver.cc:460] Webserver started at http://127.0.0.1:35307/ using document root <none> and password file <none>
I20260924 23:10:27.321768 21797 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:27.321832 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:27.322002 21797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:27.323073 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "151cdb558fd9463a9a5283e70c5f6398"
format_stamp: "Formatted at 2026-09-24 23:10:27 on dist-test-slave-ph9j"
server_key: "c706f8c527fd5cb237fd437c8492b62f"
server_key_iv: "66c2a7b8c833bc23d0e56fe0d0b6d740"
server_key_version: "encryptionkey@0"
I20260924 23:10:27.323436 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "151cdb558fd9463a9a5283e70c5f6398"
format_stamp: "Formatted at 2026-09-24 23:10:27 on dist-test-slave-ph9j"
server_key: "c706f8c527fd5cb237fd437c8492b62f"
server_key_iv: "66c2a7b8c833bc23d0e56fe0d0b6d740"
server_key_version: "encryptionkey@0"
I20260924 23:10:27.324762 21797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:10:27.325466 21812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:27.325649 21797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:27.325755 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "151cdb558fd9463a9a5283e70c5f6398"
format_stamp: "Formatted at 2026-09-24 23:10:27 on dist-test-slave-ph9j"
server_key: "c706f8c527fd5cb237fd437c8492b62f"
server_key_iv: "66c2a7b8c833bc23d0e56fe0d0b6d740"
server_key_version: "encryptionkey@0"
I20260924 23:10:27.325827 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:27.329084 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:27.330976 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:27.331131 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:27.335211 21797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260924 23:10:27.335242 21864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260924 23:10:27.335579 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:27.336571 21865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:27.339026 21865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398: Bootstrap starting.
I20260924 23:10:27.339608 21865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:27.339888 21865 log.cc:824] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:27.340682 21865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398: No bootstrap required, opened a new log
I20260924 23:10:27.342703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21797
I20260924 23:10:27.342864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:27.342773 21865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151cdb558fd9463a9a5283e70c5f6398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260924 23:10:27.342998 21865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:27.343050 21865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151cdb558fd9463a9a5283e70c5f6398, State: Initialized, Role: FOLLOWER
I20260924 23:10:27.343102 392 external_mini_cluster.cc:1428] Setting key ed2cd2ef0dd776981dd76956aeb89c05
I20260924 23:10:27.343218 21865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151cdb558fd9463a9a5283e70c5f6398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260924 23:10:27.343339 21865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:27.343390 21865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:27.343465 21865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:27.345136 21865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151cdb558fd9463a9a5283e70c5f6398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260924 23:10:27.345255 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:27.343650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:58327 (local address 127.0.0.1:39121)
0924 23:10:27.343855 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:27.343864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:27.343890 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:10:27.343983 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:27.343988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:27.344010 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:27.344120 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:27.344763 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:27.344766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:27.344767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:27.344770 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:27.344806 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:27.344817 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:27.344871 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:27.344959 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 23:10:27.345018 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260924 23:10:27.345263 21865 leader_election.cc:302] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 151cdb558fd9463a9a5283e70c5f6398; no voters:
I20260924 23:10:27.345491 21865 leader_election.cc:288] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:27.345534 21870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:27.345692 21870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [term 1 LEADER]: Becoming Leader. State: Replica: 151cdb558fd9463a9a5283e70c5f6398, State: Running, Role: LEADER
I20260924 23:10:27.345844 21865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:27.345829 21870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151cdb558fd9463a9a5283e70c5f6398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260924 23:10:27.346246 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:27.346354 21871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 151cdb558fd9463a9a5283e70c5f6398. Latest consensus state: current_term: 1 leader_uuid: "151cdb558fd9463a9a5283e70c5f6398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151cdb558fd9463a9a5283e70c5f6398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260924 23:10:27.346414 21871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:27.346544 21872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "151cdb558fd9463a9a5283e70c5f6398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151cdb558fd9463a9a5283e70c5f6398" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260924 23:10:27.346580 21872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:27.346668 21876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:27.347184 21876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:27.348721 21876 catalog_manager.cc:1347] Generated new cluster ID: 1699757f3f204397a362af627446177d
I20260924 23:10:27.348776 21876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:27.351017 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:27.348060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:35927 (local address 127.0.0.1:39121)
0924 23:10:27.348111 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:27.348113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:27.350082 (+ 1969us) server_negotiation.cc:407] Connection header received
0924 23:10:27.350151 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:27.350152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:27.350163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:27.350227 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:27.350824 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:27.350824 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:27.350825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:27.350828 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:27.350840 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:27.350843 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:27.350884 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:27.350946 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 23:10:27.350958 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 23:10:27.357174 21876 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:27.357337 21876 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:27.367552 21876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 151cdb558fd9463a9a5283e70c5f6398: Generated new TSK 0
I20260924 23:10:27.367767 21876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:27.370675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21797
2026-09-24T23:10:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:27 dist-test-slave-ph9j krb5kdc[21896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:27 dist-test-slave-ph9j krb5kdc[21896](info): set up 2 sockets
Sep 24 23:10:27 dist-test-slave-ph9j krb5kdc[21896](info): commencing operation
krb5kdc: starting...
W20260924 23:10:28.422094 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:28 dist-test-slave-ph9j krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:28.541006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:28.620826 21909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:28.621040 21909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:28.621064 21909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:28.622176 21909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:28.622211 21909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:28.622225 21909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:28.622237 21909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:28.622249 21909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:28.623629 21909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36673
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:28.623816 21909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:28.623982 21909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:28.629004 21917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:28.629068 21909 server_base.cc:1034] running on GCE node
W20260924 23:10:28.629014 21915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:28.629040 21914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:28.629416 21909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:28.629702 21909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:28.630909 21909 hybrid_clock.cc:648] HybridClock initialized: now 1790291428630885 us; error 55 us; skew 500 ppm
Sep 24 23:10:28 dist-test-slave-ph9j krb5kdc[21896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:28.633944 21909 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:28.634507 21909 webserver.cc:460] Webserver started at http://127.0.98.62:38121/ using document root <none> and password file <none>
I20260924 23:10:28.634794 21909 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:28.634866 21909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:28.635016 21909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:28.636446 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "da274b5b7eac4402969607c0fab55c05"
format_stamp: "Formatted at 2026-09-24 23:10:28 on dist-test-slave-ph9j"
server_key: "62640e7d9cc4952d900db85520d4c035"
server_key_iv: "fad93e444d72567fd0c738d497721c33"
server_key_version: "encryptionkey@0"
I20260924 23:10:28.636986 21909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "da274b5b7eac4402969607c0fab55c05"
format_stamp: "Formatted at 2026-09-24 23:10:28 on dist-test-slave-ph9j"
server_key: "62640e7d9cc4952d900db85520d4c035"
server_key_iv: "fad93e444d72567fd0c738d497721c33"
server_key_version: "encryptionkey@0"
I20260924 23:10:28.638927 21909 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 23:10:28.639801 21924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:28.640012 21909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:28.640132 21909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "da274b5b7eac4402969607c0fab55c05"
format_stamp: "Formatted at 2026-09-24 23:10:28 on dist-test-slave-ph9j"
server_key: "62640e7d9cc4952d900db85520d4c035"
server_key_iv: "fad93e444d72567fd0c738d497721c33"
server_key_version: "encryptionkey@0"
I20260924 23:10:28.640223 21909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:28.654193 21909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:28.656894 21909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:28.657039 21909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:28.661497 21909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36673
I20260924 23:10:28.661556 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36673 every 8 connection(s)
I20260924 23:10:28.661871 21909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:28.662797 21977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:28.665115 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05: Bootstrap starting.
I20260924 23:10:28.665673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 21909
I20260924 23:10:28.665792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:28.665792 21977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:28.666040 392 external_mini_cluster.cc:1428] Setting key 484e2457b6eebf07ba27927f0afeea1f
I20260924 23:10:28.666179 21977 log.cc:824] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:28.666936 21977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05: No bootstrap required, opened a new log
I20260924 23:10:28.669996 21977 raft_consensus.cc:348] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da274b5b7eac4402969607c0fab55c05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36673 } }
I20260924 23:10:28.670172 21977 raft_consensus.cc:374] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:28.670220 21977 raft_consensus.cc:729] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da274b5b7eac4402969607c0fab55c05, State: Initialized, Role: FOLLOWER
I20260924 23:10:28.670358 21977 consensus_queue.cc:260] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da274b5b7eac4402969607c0fab55c05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36673 } }
I20260924 23:10:28.670451 21977 raft_consensus.cc:388] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:28.670495 21977 raft_consensus.cc:482] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:28.670565 21977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:28.671344 21977 raft_consensus.cc:504] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da274b5b7eac4402969607c0fab55c05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36673 } }
I20260924 23:10:28.671447 21977 leader_election.cc:302] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da274b5b7eac4402969607c0fab55c05; no voters:
I20260924 23:10:28.671640 21977 leader_election.cc:288] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:28.671722 21982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:28.671881 21977 sys_catalog.cc:564] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:28.671980 21982 raft_consensus.cc:686] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [term 1 LEADER]: Becoming Leader. State: Replica: da274b5b7eac4402969607c0fab55c05, State: Running, Role: LEADER
I20260924 23:10:28.672135 21982 consensus_queue.cc:237] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da274b5b7eac4402969607c0fab55c05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36673 } }
I20260924 23:10:28.672578 21983 sys_catalog.cc:455] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da274b5b7eac4402969607c0fab55c05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da274b5b7eac4402969607c0fab55c05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36673 } } }
I20260924 23:10:28.672652 21983 sys_catalog.cc:458] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:28.672812 21984 sys_catalog.cc:455] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da274b5b7eac4402969607c0fab55c05. Latest consensus state: current_term: 1 leader_uuid: "da274b5b7eac4402969607c0fab55c05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da274b5b7eac4402969607c0fab55c05" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36673 } } }
I20260924 23:10:28.672871 21984 sys_catalog.cc:458] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05 [sys.catalog]: This master's current role is: LEADER
Sep 24 23:10:28 dist-test-slave-ph9j krb5kdc[21896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260924 23:10:28.673903 21997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:28.674350 21997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:28.674477 21991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:28.674623 21991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:28.676117 21991 catalog_manager.cc:1347] Generated new cluster ID: 20c4e12f756c4ddda88e9efeeb41b46b
I20260924 23:10:28.676163 21991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:28.678087 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:28.666586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36357 (local address 127.0.98.62:36673)
0924 23:10:28.666755 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:28.666765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:28.666799 (+ 34us) server_negotiation.cc:407] Connection header received
0924 23:10:28.667446 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:28.667453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:28.667505 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:28.667649 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:28.668291 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:28.669172 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:28.670037 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:28.670652 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:28.670757 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:28.673702 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:28.673706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:28.673706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:28.673718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:28.675896 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:28.676375 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:28.676378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:28.676378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:28.676477 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:28.676767 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:28.676768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:28.676769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:28.677061 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:28.677214 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:28.677688 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 23:10:28.677801 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260924 23:10:28.682869 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:28.688472 21991 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:28.689229 21991 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:28.693691 21991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da274b5b7eac4402969607c0fab55c05: Generated new TSK 0
I20260924 23:10:28.693856 21991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:28.699110 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:28.684329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33943 (local address 127.0.98.62:36673)
0924 23:10:28.684360 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:28.684362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:28.692899 (+ 8537us) server_negotiation.cc:407] Connection header received
0924 23:10:28.692975 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:28.692976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:28.692987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:28.693030 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:28.693621 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:28.694278 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:28.694996 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:28.695240 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:28.695314 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:28.696577 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:28.696579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:28.696580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:28.696587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:28.697967 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:28.698367 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:28.698368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:28.698368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:28.698424 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:28.698634 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:28.698636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:28.698636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:28.698735 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:28.698839 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:28.698955 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 23:10:28.699016 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 23:10:28.700825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 21909
W20260924 23:10:28.704838 21894 connection.cc:535] client connection to 127.0.98.62:36673 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36673): Connection reset by peer (error 104)
2026-09-24T23:10:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:28 dist-test-slave-ph9j krb5kdc[22008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:28 dist-test-slave-ph9j krb5kdc[22008](info): set up 2 sockets
Sep 24 23:10:28 dist-test-slave-ph9j krb5kdc[22008](info): commencing operation
krb5kdc: starting...
W20260924 23:10:29.721992 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:29 dist-test-slave-ph9j krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:29.840876 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45277
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:29.919706 22021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:29.919950 22021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:29.920007 22021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:29.921784 22021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:29.921851 22021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:29.921876 22021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:29.921939 22021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:29.921970 22021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:29.923503 22021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:29.923780 22021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:29.924017 22021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:29.928803 22026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:29.928818 22027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:29.928869 22021 server_base.cc:1034] running on GCE node
W20260924 23:10:29.928802 22029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:29.929280 22021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:29.929510 22021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:29.930649 22021 hybrid_clock.cc:648] HybridClock initialized: now 1790291429930628 us; error 41 us; skew 500 ppm
Sep 24 23:10:29 dist-test-slave-ph9j krb5kdc[22008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:29.933013 22021 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:29.933421 22021 webserver.cc:460] Webserver started at http://127.0.98.62:43421/ using document root <none> and password file <none>
I20260924 23:10:29.933650 22021 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:29.933714 22021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:29.933856 22021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:29.934937 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "b202342ca834492ab6bb4465c1249c65"
format_stamp: "Formatted at 2026-09-24 23:10:29 on dist-test-slave-ph9j"
server_key: "f6639c8e97f9c02c0a3dd40fedfc9d5b"
server_key_iv: "b435d18a524418140681e18912b750a2"
server_key_version: "encryptionkey@0"
I20260924 23:10:29.935292 22021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "b202342ca834492ab6bb4465c1249c65"
format_stamp: "Formatted at 2026-09-24 23:10:29 on dist-test-slave-ph9j"
server_key: "f6639c8e97f9c02c0a3dd40fedfc9d5b"
server_key_iv: "b435d18a524418140681e18912b750a2"
server_key_version: "encryptionkey@0"
I20260924 23:10:29.936566 22021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:10:29.937234 22036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:29.937438 22021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:29.937551 22021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "b202342ca834492ab6bb4465c1249c65"
format_stamp: "Formatted at 2026-09-24 23:10:29 on dist-test-slave-ph9j"
server_key: "f6639c8e97f9c02c0a3dd40fedfc9d5b"
server_key_iv: "b435d18a524418140681e18912b750a2"
server_key_version: "encryptionkey@0"
I20260924 23:10:29.937636 22021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:29.952736 22021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:29.955061 22021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:29.955224 22021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:29.958940 22021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45277
I20260924 23:10:29.959003 22088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45277 every 8 connection(s)
I20260924 23:10:29.959338 22021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:29.960187 22089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:29.962363 22089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65: Bootstrap starting.
I20260924 23:10:29.962854 22089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:29.963073 22089 log.cc:824] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:29.963749 22089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65: No bootstrap required, opened a new log
I20260924 23:10:29.965305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22021
I20260924 23:10:29.965426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:29.965610 392 external_mini_cluster.cc:1428] Setting key dc49b6a4bdd3ea062017fe25c7d6b771
I20260924 23:10:29.965586 22089 raft_consensus.cc:348] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b202342ca834492ab6bb4465c1249c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260924 23:10:29.965782 22089 raft_consensus.cc:374] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:29.965837 22089 raft_consensus.cc:729] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b202342ca834492ab6bb4465c1249c65, State: Initialized, Role: FOLLOWER
I20260924 23:10:29.966053 22089 consensus_queue.cc:260] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b202342ca834492ab6bb4465c1249c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260924 23:10:29.966161 22089 raft_consensus.cc:388] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:29.966210 22089 raft_consensus.cc:482] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:29.966274 22089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:29.967373 22089 raft_consensus.cc:504] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b202342ca834492ab6bb4465c1249c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260924 23:10:29.967469 22089 leader_election.cc:302] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b202342ca834492ab6bb4465c1249c65; no voters:
I20260924 23:10:29.967664 22089 leader_election.cc:288] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:29.967694 22094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:29.967810 22094 raft_consensus.cc:686] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [term 1 LEADER]: Becoming Leader. State: Replica: b202342ca834492ab6bb4465c1249c65, State: Running, Role: LEADER
I20260924 23:10:29.967890 22094 consensus_queue.cc:237] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b202342ca834492ab6bb4465c1249c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } }
I20260924 23:10:29.967974 22089 sys_catalog.cc:564] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:29.968199 22095 sys_catalog.cc:455] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b202342ca834492ab6bb4465c1249c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b202342ca834492ab6bb4465c1249c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } } }
I20260924 23:10:29.968312 22095 sys_catalog.cc:458] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:29.968189 22094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b202342ca834492ab6bb4465c1249c65. Latest consensus state: current_term: 1 leader_uuid: "b202342ca834492ab6bb4465c1249c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b202342ca834492ab6bb4465c1249c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45277 } } }
I20260924 23:10:29.968487 22094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:29.969194 22098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:29.969651 22098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:29.971486 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:29.966145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50021 (local address 127.0.98.62:45277)
0924 23:10:29.966358 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:29.966366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:29.966399 (+ 33us) server_negotiation.cc:407] Connection header received
0924 23:10:29.966504 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:29.966509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:29.966548 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:29.966668 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:29.966964 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:29.968095 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:29.969282 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:29.969966 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:29.970396 (+ 430us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:29.970817 (+ 421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:29.970821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:29.970821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:29.971024 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:29.971064 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:29.971079 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:29.971172 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:29.971252 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 23:10:29.971280 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":101,"threads_started":1}
I20260924 23:10:29.971851 22098 catalog_manager.cc:1347] Generated new cluster ID: cbfbd3aabcfc4e0cb51e60dd93d6c279
I20260924 23:10:29.971938 22098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:29.972301 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:29.976397 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:29.973582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41183 (local address 127.0.98.62:45277)
0924 23:10:29.973619 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:29.973620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:29.973768 (+ 148us) server_negotiation.cc:407] Connection header received
0924 23:10:29.973822 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:29.973823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:29.973844 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:29.973931 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:29.974065 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:29.974520 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:29.975211 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:29.975433 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:29.975513 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:29.976173 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:29.976174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:29.976175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:29.976179 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:29.976193 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:29.976197 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:29.976250 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:29.976309 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 23:10:29.976321 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:10:29.978150 22098 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:29.978704 22098 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:29.981798 22098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b202342ca834492ab6bb4465c1249c65: Generated new TSK 0
I20260924 23:10:29.981997 22098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:29.984294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22021
W20260924 23:10:29.988492 22006 connection.cc:535] client connection to 127.0.98.62:45277 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45277): Connection reset by peer (error 104)
2026-09-24T23:10:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:30 dist-test-slave-ph9j krb5kdc[22120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:30 dist-test-slave-ph9j krb5kdc[22120](info): set up 2 sockets
Sep 24 23:10:30 dist-test-slave-ph9j krb5kdc[22120](info): commencing operation
krb5kdc: starting...
W20260924 23:10:31.022143 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:31 dist-test-slave-ph9j krb5kdc[22120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:31.138888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:31.218489 22133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:31.218679 22133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:31.218701 22133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:31.219820 22133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:31.219861 22133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:31.219877 22133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:31.219888 22133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:31.219900 22133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:31.221351 22133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:31.221576 22133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:31.221822 22133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:31.226811 22141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:31.226823 22138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:31.226857 22139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:31.226840 22133 server_base.cc:1034] running on GCE node
I20260924 23:10:31.227207 22133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:31.227468 22133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:31.228605 22133 hybrid_clock.cc:648] HybridClock initialized: now 1790291431228618 us; error 55 us; skew 500 ppm
Sep 24 23:10:31 dist-test-slave-ph9j krb5kdc[22120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:31.237229 22133 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:31.237684 22133 webserver.cc:460] Webserver started at http://127.0.98.62:42479/ using document root <none> and password file <none>
I20260924 23:10:31.237937 22133 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:31.237987 22133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:31.238098 22133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:31.239123 22133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "ef56828eca1b4685af2d26fb86e01162"
format_stamp: "Formatted at 2026-09-24 23:10:31 on dist-test-slave-ph9j"
server_key: "bd20f028df6eca8652c79b2d9b19331e"
server_key_iv: "4f2b4d73382af76b72bfb437cd916875"
server_key_version: "encryptionkey@0"
I20260924 23:10:31.239413 22133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef56828eca1b4685af2d26fb86e01162"
format_stamp: "Formatted at 2026-09-24 23:10:31 on dist-test-slave-ph9j"
server_key: "bd20f028df6eca8652c79b2d9b19331e"
server_key_iv: "4f2b4d73382af76b72bfb437cd916875"
server_key_version: "encryptionkey@0"
I20260924 23:10:31.240644 22133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:31.241298 22148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:31.241446 22133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:31.241530 22133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "ef56828eca1b4685af2d26fb86e01162"
format_stamp: "Formatted at 2026-09-24 23:10:31 on dist-test-slave-ph9j"
server_key: "bd20f028df6eca8652c79b2d9b19331e"
server_key_iv: "4f2b4d73382af76b72bfb437cd916875"
server_key_version: "encryptionkey@0"
I20260924 23:10:31.241586 22133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:31.244722 22133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:31.246479 22133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:31.246579 22133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:31.250149 22133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45995
I20260924 23:10:31.250197 22200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45995 every 8 connection(s)
I20260924 23:10:31.250528 22133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:31.251494 22201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:31.253242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22133
I20260924 23:10:31.253358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:31.253567 392 external_mini_cluster.cc:1428] Setting key 970ada02f544e0ac78edb107b1331934
I20260924 23:10:31.254333 22201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162: Bootstrap starting.
I20260924 23:10:31.255013 22201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:31.255745 22201 log.cc:824] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:31.256238 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:31.254080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40727 (local address 127.0.98.62:45995)
0924 23:10:31.254225 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:31.254235 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:31.254265 (+ 30us) server_negotiation.cc:407] Connection header received
0924 23:10:31.254826 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:31.254834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:31.254866 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:31.255015 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:31.255683 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:31.255688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:31.255688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:31.255693 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:31.255735 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:31.255749 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:31.255846 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:31.255976 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 23:10:31.256022 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260924 23:10:31.256513 22201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162: No bootstrap required, opened a new log
I20260924 23:10:31.257331 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:31.258881 22201 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56828eca1b4685af2d26fb86e01162" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45995 } }
I20260924 23:10:31.259032 22201 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:31.259078 22201 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef56828eca1b4685af2d26fb86e01162, State: Initialized, Role: FOLLOWER
I20260924 23:10:31.259208 22201 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56828eca1b4685af2d26fb86e01162" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45995 } }
I20260924 23:10:31.259303 22201 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:31.259344 22201 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:31.259388 22201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:31.259720 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:31.258673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43241 (local address 127.0.98.62:45995)
0924 23:10:31.258705 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:31.258707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:31.258828 (+ 121us) server_negotiation.cc:407] Connection header received
0924 23:10:31.258900 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:31.258901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:31.258906 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:31.258956 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:31.259511 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:31.259512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:31.259512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:31.259516 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:31.259527 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:31.259531 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:31.259589 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:31.259645 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 23:10:31.259658 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 23:10:31.260028 22201 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56828eca1b4685af2d26fb86e01162" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45995 } }
I20260924 23:10:31.260135 22201 leader_election.cc:302] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef56828eca1b4685af2d26fb86e01162; no voters:
I20260924 23:10:31.260281 22201 leader_election.cc:288] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:31.260303 22211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:31.260421 22211 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [term 1 LEADER]: Becoming Leader. State: Replica: ef56828eca1b4685af2d26fb86e01162, State: Running, Role: LEADER
I20260924 23:10:31.260494 22211 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56828eca1b4685af2d26fb86e01162" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45995 } }
I20260924 23:10:31.260494 22201 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:31.261708 22213 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef56828eca1b4685af2d26fb86e01162. Latest consensus state: current_term: 1 leader_uuid: "ef56828eca1b4685af2d26fb86e01162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56828eca1b4685af2d26fb86e01162" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45995 } } }
I20260924 23:10:31.261788 22213 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:31.261967 22212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef56828eca1b4685af2d26fb86e01162" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef56828eca1b4685af2d26fb86e01162" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45995 } } }
I20260924 23:10:31.262118 22212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162 [sys.catalog]: This master's current role is: LEADER
W20260924 23:10:31.262318 22226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:31.262374 22226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:31.262492 22227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:31.262645 22227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:31.263989 22227 catalog_manager.cc:1347] Generated new cluster ID: 8f41f87a65864a07956022c173e1b196
I20260924 23:10:31.264039 22227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:31.286161 22227 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:31.286351 22227 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:31.288308 22227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef56828eca1b4685af2d26fb86e01162: Generated new TSK 0
I20260924 23:10:31.288483 22227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:31.291875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22133
2026-09-24T23:10:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:31 dist-test-slave-ph9j krb5kdc[22232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:31 dist-test-slave-ph9j krb5kdc[22232](info): set up 2 sockets
Sep 24 23:10:31 dist-test-slave-ph9j krb5kdc[22232](info): commencing operation
krb5kdc: starting...
W20260924 23:10:32.322103 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:32 dist-test-slave-ph9j krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:32.438575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:32.519596 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:32.519801 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:32.519824 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:32.521317 22245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:32.521396 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:32.521418 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:32.521440 22245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:32.521479 22245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:32.523422 22245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:32.523707 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:32.523936 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:32.529542 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:32.529573 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:32.529665 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:32.529724 22245 server_base.cc:1034] running on GCE node
I20260924 23:10:32.529999 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:32.530285 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:32.531421 22245 hybrid_clock.cc:648] HybridClock initialized: now 1790291432531399 us; error 38 us; skew 500 ppm
Sep 24 23:10:32 dist-test-slave-ph9j krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:32.533784 22245 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:32.534255 22245 webserver.cc:460] Webserver started at http://127.0.98.62:34415/ using document root <none> and password file <none>
I20260924 23:10:32.534480 22245 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:32.534552 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:32.534693 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:32.535706 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "385b7d46ecf94e26ad242eabaa649cb9"
format_stamp: "Formatted at 2026-09-24 23:10:32 on dist-test-slave-ph9j"
server_key: "1355c9f00ba121d36688e38c69c869a9"
server_key_iv: "d94aa4a4656bd43de3f76492f69eab7d"
server_key_version: "encryptionkey@0"
I20260924 23:10:32.536058 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "385b7d46ecf94e26ad242eabaa649cb9"
format_stamp: "Formatted at 2026-09-24 23:10:32 on dist-test-slave-ph9j"
server_key: "1355c9f00ba121d36688e38c69c869a9"
server_key_iv: "d94aa4a4656bd43de3f76492f69eab7d"
server_key_version: "encryptionkey@0"
I20260924 23:10:32.537324 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:10:32.538067 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:32.538230 22245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:32.538324 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "385b7d46ecf94e26ad242eabaa649cb9"
format_stamp: "Formatted at 2026-09-24 23:10:32 on dist-test-slave-ph9j"
server_key: "1355c9f00ba121d36688e38c69c869a9"
server_key_iv: "d94aa4a4656bd43de3f76492f69eab7d"
server_key_version: "encryptionkey@0"
I20260924 23:10:32.538404 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:32.548593 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:32.550765 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:32.550931 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:32.554965 22245 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34085
I20260924 23:10:32.555007 22312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34085 every 8 connection(s)
I20260924 23:10:32.555385 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:32.556209 22313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:32.558513 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9: Bootstrap starting.
I20260924 23:10:32.559089 22313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:32.559362 22313 log.cc:824] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:32.560127 22313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9: No bootstrap required, opened a new log
I20260924 23:10:32.561693 22313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385b7d46ecf94e26ad242eabaa649cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 23:10:32.561830 22313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:32.561852 22313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385b7d46ecf94e26ad242eabaa649cb9, State: Initialized, Role: FOLLOWER
I20260924 23:10:32.562093 22313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385b7d46ecf94e26ad242eabaa649cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 23:10:32.562176 22313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:32.562225 22313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:32.562276 22313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:32.562789 22313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385b7d46ecf94e26ad242eabaa649cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 23:10:32.562894 22313 leader_election.cc:302] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 385b7d46ecf94e26ad242eabaa649cb9; no voters:
I20260924 23:10:32.563068 22313 leader_election.cc:288] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:32.563098 22316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:32.563212 22316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [term 1 LEADER]: Becoming Leader. State: Replica: 385b7d46ecf94e26ad242eabaa649cb9, State: Running, Role: LEADER
I20260924 23:10:32.563232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22245
I20260924 23:10:32.563330 22313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:32.563361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:32.563347 22316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385b7d46ecf94e26ad242eabaa649cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 23:10:32.563534 392 external_mini_cluster.cc:1428] Setting key 397fe3da218b0bf94ca2c9a643e24383
I20260924 23:10:32.563696 22317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "385b7d46ecf94e26ad242eabaa649cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385b7d46ecf94e26ad242eabaa649cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } } }
I20260924 23:10:32.563777 22317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:32.563835 22318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 385b7d46ecf94e26ad242eabaa649cb9. Latest consensus state: current_term: 1 leader_uuid: "385b7d46ecf94e26ad242eabaa649cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385b7d46ecf94e26ad242eabaa649cb9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } } }
I20260924 23:10:32.563917 22318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:32.564118 22323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:32.564626 22323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:32.566258 22323 catalog_manager.cc:1347] Generated new cluster ID: 10bdbed50ebe4309942562b94534bf11
I20260924 23:10:32.566308 22323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 23:10:32 dist-test-slave-ph9j krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:32.576104 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:32.564186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52577 (local address 127.0.98.62:34085)
0924 23:10:32.564739 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:32.564749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:32.565248 (+ 499us) server_negotiation.cc:407] Connection header received
0924 23:10:32.565344 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:32.565349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:32.565393 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:32.565517 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:32.566361 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:32.567737 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:32.568863 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:32.569461 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:32.569568 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:32.572401 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:32.572405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:32.572406 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:32.572419 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:32.574469 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:32.574841 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:32.574845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:32.574845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:32.574966 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:32.575162 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:32.575163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:32.575164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:32.575495 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:32.575657 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:32.575727 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 23:10:32.575869 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":70,"threads_started":1}
I20260924 23:10:32.577031 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:32.586071 22323 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:32.586819 22323 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:32.593662 22327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:32.582233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44605 (local address 127.0.98.62:34085)
0924 23:10:32.582271 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:32.582273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:32.586075 (+ 3802us) server_negotiation.cc:407] Connection header received
0924 23:10:32.586182 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:32.586184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:32.586207 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:32.586236 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:32.586987 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:32.588143 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:32.589111 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:32.589446 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:32.589518 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:32.590939 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:32.590941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:32.590941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:32.590948 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:32.592508 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:32.592870 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:32.592873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:32.592873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:32.592942 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:32.593133 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:32.593135 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:32.593135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:32.593271 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:32.593377 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:32.593472 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 23:10:32.593566 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:32.594127 22323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 385b7d46ecf94e26ad242eabaa649cb9: Generated new TSK 0
I20260924 23:10:32.594368 22323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:32.596870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22245
2026-09-24T23:10:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:32 dist-test-slave-ph9j krb5kdc[22344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:32 dist-test-slave-ph9j krb5kdc[22344](info): set up 2 sockets
Sep 24 23:10:32 dist-test-slave-ph9j krb5kdc[22344](info): commencing operation
krb5kdc: starting...
W20260924 23:10:33.630545 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:33 dist-test-slave-ph9j krb5kdc[22344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:33.749253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:33.832588 22357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:33.832873 22357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:33.832917 22357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:33.834242 22357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:33.834316 22357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:33.834342 22357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:33.834379 22357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:33.834408 22357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:33.836018 22357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:33.836301 22357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:33.836532 22357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:33.841518 22362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:33.841506 22365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:33.841507 22363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:33.841688 22357 server_base.cc:1034] running on GCE node
I20260924 23:10:33.842059 22357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:33.842368 22357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:33.843524 22357 hybrid_clock.cc:648] HybridClock initialized: now 1790291433843504 us; error 38 us; skew 500 ppm
Sep 24 23:10:33 dist-test-slave-ph9j krb5kdc[22344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:33.846064 22357 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:33.846540 22357 webserver.cc:460] Webserver started at http://127.0.98.62:34285/ using document root <none> and password file <none>
I20260924 23:10:33.846745 22357 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:33.846788 22357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:33.846884 22357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:33.847930 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a"
format_stamp: "Formatted at 2026-09-24 23:10:33 on dist-test-slave-ph9j"
server_key: "02fcdaa712410e0647f0bee7ae0d8430"
server_key_iv: "b5d83f4dbc5677575a08bbb0b06f189a"
server_key_version: "encryptionkey@0"
I20260924 23:10:33.848241 22357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a"
format_stamp: "Formatted at 2026-09-24 23:10:33 on dist-test-slave-ph9j"
server_key: "02fcdaa712410e0647f0bee7ae0d8430"
server_key_iv: "b5d83f4dbc5677575a08bbb0b06f189a"
server_key_version: "encryptionkey@0"
I20260924 23:10:33.849543 22357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:33.850354 22372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:33.850546 22357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:33.850660 22357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a"
format_stamp: "Formatted at 2026-09-24 23:10:33 on dist-test-slave-ph9j"
server_key: "02fcdaa712410e0647f0bee7ae0d8430"
server_key_iv: "b5d83f4dbc5677575a08bbb0b06f189a"
server_key_version: "encryptionkey@0"
I20260924 23:10:33.850742 22357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:33.860905 22357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:33.863266 22357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:33.863428 22357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:33.867513 22357 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39877
I20260924 23:10:33.867544 22424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39877 every 8 connection(s)
I20260924 23:10:33.867897 22357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:33.868742 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:33.871060 22425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a: Bootstrap starting.
I20260924 23:10:33.871582 22425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:33.871809 22425 log.cc:824] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:33.872431 22425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a: No bootstrap required, opened a new log
I20260924 23:10:33.873903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22357
I20260924 23:10:33.874025 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:33.874090 22425 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260924 23:10:33.874234 392 external_mini_cluster.cc:1428] Setting key 28d6f08d386b242c6dda94cd8427ae1a
I20260924 23:10:33.874254 22425 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:33.874279 22425 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc962bd75ce7440d9faf9fbfd8b12b4a, State: Initialized, Role: FOLLOWER
I20260924 23:10:33.874442 22425 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260924 23:10:33.874531 22425 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:33.874553 22425 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:33.874621 22425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:33.875746 22425 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260924 23:10:33.875855 22425 leader_election.cc:302] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc962bd75ce7440d9faf9fbfd8b12b4a; no voters:
I20260924 23:10:33.876011 22425 leader_election.cc:288] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:33.876087 22430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:33.876255 22425 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:33.876253 22430 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [term 1 LEADER]: Becoming Leader. State: Replica: fc962bd75ce7440d9faf9fbfd8b12b4a, State: Running, Role: LEADER
I20260924 23:10:33.876454 22430 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260924 23:10:33.876842 22430 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc962bd75ce7440d9faf9fbfd8b12b4a. Latest consensus state: current_term: 1 leader_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260924 23:10:33.877070 22430 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:33.876991 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc962bd75ce7440d9faf9fbfd8b12b4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260924 23:10:33.878048 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a [sys.catalog]: This master's current role is: LEADER
W20260924 23:10:33.878115 22445 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:33.878206 22445 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:33.878268 22446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:33.878409 22446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:33.880316 22446 catalog_manager.cc:1347] Generated new cluster ID: e5967fa34c534ac68ded65dfaffe4351
I20260924 23:10:33.880378 22446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:33.880501 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:33.874889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49163 (local address 127.0.98.62:39877)
0924 23:10:33.875065 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:33.875074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:33.875110 (+ 36us) server_negotiation.cc:407] Connection header received
0924 23:10:33.875261 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:33.875268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:33.875337 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:33.875501 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:33.875680 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:33.877075 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:33.878226 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:33.878796 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:33.878907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:33.879667 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:33.879670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:33.879671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:33.879850 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:33.879879 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:33.879889 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:33.879941 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:33.880160 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 23:10:33.880225 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260924 23:10:33.881376 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:33.889386 22429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:33.882719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44717 (local address 127.0.98.62:39877)
0924 23:10:33.882749 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:33.882751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:33.886160 (+ 3409us) server_negotiation.cc:407] Connection header received
0924 23:10:33.886221 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:33.886222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:33.886232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:33.886288 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:33.886410 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:33.887127 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:33.888151 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:33.888430 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:33.888489 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:33.889164 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:33.889165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:33.889165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:33.889169 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:33.889182 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:33.889186 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:33.889234 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:33.889294 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 23:10:33.889316 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 23:10:33.907063 22446 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:33.907804 22446 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:33.911767 22446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc962bd75ce7440d9faf9fbfd8b12b4a: Generated new TSK 0
I20260924 23:10:33.911965 22446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:33.913885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22357
2026-09-24T23:10:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:33 dist-test-slave-ph9j krb5kdc[22456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:33 dist-test-slave-ph9j krb5kdc[22456](info): set up 2 sockets
Sep 24 23:10:33 dist-test-slave-ph9j krb5kdc[22456](info): commencing operation
krb5kdc: starting...
W20260924 23:10:34.958226 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:35 dist-test-slave-ph9j krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:35.076893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:35.156215 22469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:35.156440 22469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:35.156463 22469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:35.157648 22469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:35.157692 22469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:35.157706 22469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:35.157755 22469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:35.157783 22469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:35.159260 22469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:35.159528 22469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:35.159759 22469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:35.164598 22474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:35.164598 22475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:35.164605 22469 server_base.cc:1034] running on GCE node
W20260924 23:10:35.164605 22477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:35.165060 22469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:35.165377 22469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:35.166546 22469 hybrid_clock.cc:648] HybridClock initialized: now 1790291435166519 us; error 39 us; skew 500 ppm
Sep 24 23:10:35 dist-test-slave-ph9j krb5kdc[22456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:35.168949 22469 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:35.169440 22469 webserver.cc:460] Webserver started at http://127.0.98.62:37065/ using document root <none> and password file <none>
I20260924 23:10:35.169665 22469 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:35.169731 22469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:35.169924 22469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:35.170989 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "50fb782a1af04b389a98c0dce21fa15d"
format_stamp: "Formatted at 2026-09-24 23:10:35 on dist-test-slave-ph9j"
server_key: "68a7a4bb6549829af052f6bf31a44816"
server_key_iv: "087513a39370fd7f684b260231e6e5a9"
server_key_version: "encryptionkey@0"
I20260924 23:10:35.171355 22469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "50fb782a1af04b389a98c0dce21fa15d"
format_stamp: "Formatted at 2026-09-24 23:10:35 on dist-test-slave-ph9j"
server_key: "68a7a4bb6549829af052f6bf31a44816"
server_key_iv: "087513a39370fd7f684b260231e6e5a9"
server_key_version: "encryptionkey@0"
I20260924 23:10:35.172652 22469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 23:10:35.173337 22484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:35.173509 22469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 23:10:35.173606 22469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "50fb782a1af04b389a98c0dce21fa15d"
format_stamp: "Formatted at 2026-09-24 23:10:35 on dist-test-slave-ph9j"
server_key: "68a7a4bb6549829af052f6bf31a44816"
server_key_iv: "087513a39370fd7f684b260231e6e5a9"
server_key_version: "encryptionkey@0"
I20260924 23:10:35.173686 22469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:35.176951 22469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:35.178845 22469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:35.179003 22469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:35.183054 22469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42957
I20260924 23:10:35.183079 22536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42957 every 8 connection(s)
I20260924 23:10:35.183406 22469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:35.184374 22537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:35.186677 22537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d: Bootstrap starting.
I20260924 23:10:35.187331 22537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:35.187609 22537 log.cc:824] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:35.188257 22537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d: No bootstrap required, opened a new log
I20260924 23:10:35.189798 22537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fb782a1af04b389a98c0dce21fa15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42957 } }
I20260924 23:10:35.190013 22537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 23:10:35.190062 22537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fb782a1af04b389a98c0dce21fa15d, State: Initialized, Role: FOLLOWER
I20260924 23:10:35.190229 22537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fb782a1af04b389a98c0dce21fa15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42957 } }
I20260924 23:10:35.190287 22537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 23:10:35.190346 22537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 23:10:35.190397 22537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:35.190905 22537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fb782a1af04b389a98c0dce21fa15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42957 } }
I20260924 23:10:35.191020 22537 leader_election.cc:302] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50fb782a1af04b389a98c0dce21fa15d; no voters:
I20260924 23:10:35.191205 22537 leader_election.cc:288] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 23:10:35.191238 22540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:35.191416 22540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [term 1 LEADER]: Becoming Leader. State: Replica: 50fb782a1af04b389a98c0dce21fa15d, State: Running, Role: LEADER
I20260924 23:10:35.191459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22469
I20260924 23:10:35.191483 22537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:35.191571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:35.191619 22540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fb782a1af04b389a98c0dce21fa15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42957 } }
I20260924 23:10:35.191783 392 external_mini_cluster.cc:1428] Setting key 428d8e914f63a8b0da78dc951b8e623c
I20260924 23:10:35.192291 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50fb782a1af04b389a98c0dce21fa15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fb782a1af04b389a98c0dce21fa15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42957 } } }
I20260924 23:10:35.192561 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:35.192478 22541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50fb782a1af04b389a98c0dce21fa15d. Latest consensus state: current_term: 1 leader_uuid: "50fb782a1af04b389a98c0dce21fa15d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50fb782a1af04b389a98c0dce21fa15d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42957 } } }
I20260924 23:10:35.192721 22541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:35.192829 22551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:35.193334 22551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:35.194736 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:35.192274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53233 (local address 127.0.98.62:42957)
0924 23:10:35.192958 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:35.192968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:35.193000 (+ 32us) server_negotiation.cc:407] Connection header received
0924 23:10:35.193172 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:35.193179 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:35.193212 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:35.193355 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:35.194303 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:35.194306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:35.194306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:35.194309 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:35.194338 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:35.194348 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:35.194428 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:35.194504 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 23:10:35.194543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":76,"threads_started":1}
I20260924 23:10:35.195294 22551 catalog_manager.cc:1347] Generated new cluster ID: 7b92756fe65b4a8983236caaad1dd1da
I20260924 23:10:35.195348 22551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:35.195600 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:35.198206 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:35.196987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53991 (local address 127.0.98.62:42957)
0924 23:10:35.197014 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:35.197016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:35.197207 (+ 191us) server_negotiation.cc:407] Connection header received
0924 23:10:35.197285 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:35.197286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:35.197308 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:35.197333 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:35.197990 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:35.197991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:35.197992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 23:10:35.197995 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:35.198007 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 23:10:35.198011 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:35.198039 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:35.198125 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 23:10:35.198136 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260924 23:10:35.203738 22551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:35.203904 22551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:35.206655 22551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50fb782a1af04b389a98c0dce21fa15d: Generated new TSK 0
I20260924 23:10:35.206822 22551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:35.213987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22469
2026-09-24T23:10:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1298 ms)
[----------] 18 tests from AuthTokenIssuingTest (23777 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:35 dist-test-slave-ph9j krb5kdc[22568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:35 dist-test-slave-ph9j krb5kdc[22568](info): set up 2 sockets
Sep 24 23:10:35 dist-test-slave-ph9j krb5kdc[22568](info): commencing operation
krb5kdc: starting...
W20260924 23:10:36.242007 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:36.356905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44437
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44437,127.0.98.61:39435,127.0.98.60:34499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:36.437039 22581 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:10:36.437238 22581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:36.437256 22581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:36.437275 22581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:36.438429 22581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:36.438472 22581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:36.438486 22581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:36.438498 22581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:36.438510 22581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:36.440333 22581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44437,127.0.98.61:39435,127.0.98.60:34499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:36.440574 22581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:36.440819 22581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:36.445436 22586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:36.445499 22581 server_base.cc:1034] running on GCE node
W20260924 23:10:36.445643 22587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:36.445578 22589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:36.446036 22581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:36.446296 22581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:36.447453 22581 hybrid_clock.cc:648] HybridClock initialized: now 1790291436447425 us; error 35 us; skew 500 ppm
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:36.453393 22581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:36.453847 22581 webserver.cc:460] Webserver started at http://127.0.98.62:34891/ using document root <none> and password file <none>
I20260924 23:10:36.454111 22581 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:36.454182 22581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:36.454339 22581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:36.455389 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "99436a9dee524bb884134ce6e8e06271"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "3ebb168ca63a68f9faae4f178d6ea51e"
server_key_iv: "1d95a1610dc7327f4d7547dea378dd53"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.455739 22581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "99436a9dee524bb884134ce6e8e06271"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "3ebb168ca63a68f9faae4f178d6ea51e"
server_key_iv: "1d95a1610dc7327f4d7547dea378dd53"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.457034 22581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 23:10:36.457787 22596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:36.457990 22581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 23:10:36.458106 22581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "99436a9dee524bb884134ce6e8e06271"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "3ebb168ca63a68f9faae4f178d6ea51e"
server_key_iv: "1d95a1610dc7327f4d7547dea378dd53"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.458187 22581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:36.473537 22581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:36.475543 22581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:36.475700 22581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:36.479578 22581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44437
I20260924 23:10:36.479612 22648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44437 every 8 connection(s)
I20260924 23:10:36.479981 22581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:36.480959 22649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:36.481560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22581
I20260924 23:10:36.481645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:36.481868 392 external_mini_cluster.cc:1428] Setting key 14913ca68c1042d3d084653da7448f34
I20260924 23:10:36.482688 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 23:10:36.492496 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.483404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44437 (local address 127.0.0.1:40092)
0924 23:10:36.483615 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.483657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.484286 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.484545 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.484547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.484829 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.484984 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.484989 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.486123 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.486124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.486787 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.486789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.487285 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.487286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.487360 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.487888 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.487897 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.489243 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.490915 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.490921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.490923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.491144 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.491375 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.491376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.491376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.491506 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.491909 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.491914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.491980 (+ 66us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.492078 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.492306 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":93,"thread_start_us":84,"threads_started":1}
I20260924 23:10:36.492497 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.483426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40092 (local address 127.0.98.62:44437)
0924 23:10:36.483577 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.483586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.483758 (+ 172us) server_negotiation.cc:407] Connection header received
0924 23:10:36.484418 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.484419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.484435 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.484540 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.485091 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.486028 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.486876 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.487175 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.487293 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.489329 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.489336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.489337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.489343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.490776 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.491236 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.491239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.491239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.491314 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.491574 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.491575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.491575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.491835 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.491933 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.492099 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.492245 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260924 23:10:36.493335 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:36.493705 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.493562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37228)
0924 23:10:36.493612 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.493673 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:10:36.493854 22597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
I20260924 23:10:36.494297 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.494208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37236)
0924 23:10:36.494256 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.494280 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 23:10:36.494402 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39435: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 23:10:36.507082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39435
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44437,127.0.98.61:39435,127.0.98.60:34499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:10:36.543641 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } attempt: 1
I20260924 23:10:36.544173 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.544031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37250)
0924 23:10:36.544088 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.544137 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:10:36.544541 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.544454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37266)
0924 23:10:36.544494 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.544521 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 23:10:36.544654 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39435: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
W20260924 23:10:36.590760 22656 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:10:36.590965 22656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:36.590981 22656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:36.591028 22656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:36.592159 22656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:36.592197 22656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:36.592211 22656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:36.592224 22656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:36.592274 22656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:36.593752 22656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44437,127.0.98.61:39435,127.0.98.60:34499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:36.594022 22656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:36.594254 22656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:10:36.599135 22656 server_base.cc:1034] running on GCE node
W20260924 23:10:36.599138 22664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:36.599280 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:36.599140 22662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:36.599639 22656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:36.599948 22656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:36.601117 22656 hybrid_clock.cc:648] HybridClock initialized: now 1790291436601102 us; error 50 us; skew 500 ppm
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:36.603555 22656 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 23:10:36.604023 22656 webserver.cc:460] Webserver started at http://127.0.98.61:34075/ using document root <none> and password file <none>
I20260924 23:10:36.604260 22656 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:36.604328 22656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:36.604477 22656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:36.605509 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data/instance:
uuid: "22d6be21866a4f1080497f76f50de1f3"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "be8ef4e95d3763674061c83c14559f3d"
server_key_iv: "65c421e737160a8e504a1d4a257cc0c6"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.605859 22656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal/instance:
uuid: "22d6be21866a4f1080497f76f50de1f3"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "be8ef4e95d3763674061c83c14559f3d"
server_key_iv: "65c421e737160a8e504a1d4a257cc0c6"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.607155 22656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:36.607865 22671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:36.608019 22656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:36.608114 22656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal
uuid: "22d6be21866a4f1080497f76f50de1f3"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "be8ef4e95d3763674061c83c14559f3d"
server_key_iv: "65c421e737160a8e504a1d4a257cc0c6"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.608193 22656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:36.612829 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } attempt: 2
I20260924 23:10:36.613201 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.613068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37276)
0924 23:10:36.613129 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.613168 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 23:10:36.613538 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.613456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37280)
0924 23:10:36.613497 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.613520 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:10:36.613626 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39435: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39435: connect: Connection refused (error 111)
I20260924 23:10:36.618441 22656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:36.620115 22656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:36.620272 22656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:36.624147 22656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39435
I20260924 23:10:36.624176 22723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39435 every 8 connection(s)
I20260924 23:10:36.624532 22656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
I20260924 23:10:36.625437 22724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:36.626948 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:36.631414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22656
I20260924 23:10:36.631536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-1/wal/instance
I20260924 23:10:36.631747 392 external_mini_cluster.cc:1428] Setting key 94a4dec3771d494d6a4be2163e7fb517
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:36.635859 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.627618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.98.62:44437)
0924 23:10:36.627693 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.627695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.627846 (+ 151us) server_negotiation.cc:407] Connection header received
0924 23:10:36.628323 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.628324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.628336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.628394 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.629027 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.629496 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.630590 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.630780 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.630833 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.633093 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.633094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.633095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.633103 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.634471 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.634897 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.634898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.634899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.634957 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.635203 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.635204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.635205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.635320 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.635393 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.635665 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.635752 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 23:10:36.636044 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.627527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44437 (local address 127.0.0.1:40096)
0924 23:10:36.627729 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.627764 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.628241 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.628469 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.628472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.628754 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.628946 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.628951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.629595 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.629596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.630484 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.630487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.630862 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.630863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.630964 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.631478 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.631490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.632991 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.634564 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.634571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.634573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.634799 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.635022 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.635023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.635024 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.635134 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.635386 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.635392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.635536 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.635646 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.635804 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":61,"threads_started":1}
I20260924 23:10:36.636759 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:10:36.645444 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.637039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37294)
0924 23:10:36.637241 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.637253 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.637304 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.637446 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.637446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.637585 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.637628 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.637629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.638568 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.638569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.639130 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.639132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.639618 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.639619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.639681 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.640100 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.640105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.642573 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.644382 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.644383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.644383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.644535 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.644762 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.644764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.644764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.644817 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.645115 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.645116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.645153 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.645225 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.645332 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260924 23:10:36.645629 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.637094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.0.98.61:39435)
0924 23:10:36.637250 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.637256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.637302 (+ 46us) server_negotiation.cc:407] Connection header received
0924 23:10:36.637368 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.637371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.637384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.637447 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.637710 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.638477 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.639216 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.639534 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.639618 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.642656 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.642663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.642663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.642669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.644309 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.644635 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.644639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.644639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.644698 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.644892 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.644893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.644894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.645047 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.645138 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.645254 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.645446 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260924 23:10:36.646296 22724 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:36.646661 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.646543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58044)
0924 23:10:36.646589 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.646633 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:10:36.646813 22675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
I20260924 23:10:36.647235 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.647146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58048)
0924 23:10:36.647191 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.647214 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 23:10:36.647347 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34499: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 23:10:36.657303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34499
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44437,127.0.98.61:39435,127.0.98.60:34499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:10:36.696564 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } attempt: 1
I20260924 23:10:36.697089 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.696956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58058)
0924 23:10:36.697017 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.697059 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 23:10:36.697463 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.697368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58062)
0924 23:10:36.697412 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.697437 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:10:36.697548 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34499: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
I20260924 23:10:36.704789 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } attempt: 3
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:10:36.712095 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.705142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37300)
0924 23:10:36.705186 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.705204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.705269 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.705429 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.705430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.705592 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.705666 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.705667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.706425 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.706427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.706990 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.706991 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.707395 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.707396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.707475 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.707964 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.707968 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.709212 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.711124 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.711125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.711125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.711292 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.711528 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.711529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.711530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.711586 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.711856 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.711857 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.711900 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.711954 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.712029 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 23:10:36.712167 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.705202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.0.98.61:39435)
0924 23:10:36.705264 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.705267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.705278 (+ 11us) server_negotiation.cc:407] Connection header received
0924 23:10:36.705340 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.705341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.705354 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.705422 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.705747 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.706330 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.707090 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.707317 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.707399 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.709296 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.709297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.709298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.709306 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.711048 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.711367 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.711368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.711369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.711464 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.711650 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.711651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.711651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.711794 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.711860 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.711997 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.712077 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 23:10:36.712577 22649 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:36.712945 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.712848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58068)
0924 23:10:36.712893 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.712922 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 23:10:36.713258 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.713169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58072)
0924 23:10:36.713216 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.713239 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:10:36.713377 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34499: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
W20260924 23:10:36.739811 22731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:10:36.740015 22731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:36.740032 22731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:36.740080 22731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:36.741176 22731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:36.741214 22731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:36.741228 22731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:36.741276 22731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:36.741308 22731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:36.742865 22731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44437,127.0.98.61:39435,127.0.98.60:34499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:36.743109 22731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:36.743351 22731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 23:10:36.744522 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } attempt: 1
I20260924 23:10:36.744971 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.744829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58074)
0924 23:10:36.744903 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.744943 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 23:10:36.745299 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.745215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58078)
0924 23:10:36.745255 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.745279 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:10:36.745406 22649 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34499: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
W20260924 23:10:36.748353 22736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:36.748360 22737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:36.748477 22731 server_base.cc:1034] running on GCE node
W20260924 23:10:36.748360 22739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:36.748865 22731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:36.749107 22731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:36.750237 22731 hybrid_clock.cc:648] HybridClock initialized: now 1790291436750218 us; error 34 us; skew 500 ppm
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:36.752544 22731 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 23:10:36.752959 22731 webserver.cc:460] Webserver started at http://127.0.98.60:39515/ using document root <none> and password file <none>
I20260924 23:10:36.753170 22731 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:36.753213 22731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:36.753360 22731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:36.754456 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data/instance:
uuid: "501a5784ed244207831cd94ce3be793c"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "2c1592d5e52f4b5b28d20908f0abf23c"
server_key_iv: "f4127dbb98c04a59d3624d094cc360d7"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.754814 22731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal/instance:
uuid: "501a5784ed244207831cd94ce3be793c"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "2c1592d5e52f4b5b28d20908f0abf23c"
server_key_iv: "f4127dbb98c04a59d3624d094cc360d7"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.756114 22731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:36.756915 22746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:36.757118 22731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 23:10:36.757225 22731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal
uuid: "501a5784ed244207831cd94ce3be793c"
format_stamp: "Formatted at 2026-09-24 23:10:36 on dist-test-slave-ph9j"
server_key: "2c1592d5e52f4b5b28d20908f0abf23c"
server_key_iv: "f4127dbb98c04a59d3624d094cc360d7"
server_key_version: "encryptionkey@0"
I20260924 23:10:36.757305 22731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:36.765683 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } attempt: 2
I20260924 23:10:36.766156 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.766033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58094)
0924 23:10:36.766089 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.766128 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:10:36.766551 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.766431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58100)
0924 23:10:36.766470 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.766524 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 23:10:36.766727 22724 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34499: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34499: connect: Connection refused (error 111)
I20260924 23:10:36.781090 22731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:36.783212 22731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:36.783383 22731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:36.787437 22731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34499
I20260924 23:10:36.787472 22798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34499 every 8 connection(s)
I20260924 23:10:36.787842 22731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
I20260924 23:10:36.788842 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:36.790586 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:36.791921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22731
I20260924 23:10:36.792032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790291169445806-392-0/minicluster-data/master-2/wal/instance
I20260924 23:10:36.792243 392 external_mini_cluster.cc:1428] Setting key 063fb8ffcf05617102f82322da81d816
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:10:36.800814 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.791458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.0.98.62:44437)
0924 23:10:36.791533 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.791534 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.791752 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:10:36.792398 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.792400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.792412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.792487 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.793221 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.793724 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.794689 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.794881 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.794954 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.797363 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.797365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.797365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.797374 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.799082 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.799623 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.799623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.799624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.799674 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.799975 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.799976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.799976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.800147 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.800229 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.800625 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.800714 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:10:36.801347 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.791449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44437 (local address 127.0.0.1:40108)
0924 23:10:36.791629 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.791667 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.792251 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.792572 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.792575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.792930 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.793124 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.793128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.793818 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.793819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.794602 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.794604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.794991 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.794993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.795105 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.795628 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.795636 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.797214 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.799183 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.799191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.799194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.799548 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.799736 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.799736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.799737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.799874 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.800304 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.800315 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.800462 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.800932 (+ 470us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.801101 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 23:10:36.801693 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.792665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.0.98.60:34499)
0924 23:10:36.792852 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.792857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.792884 (+ 27us) server_negotiation.cc:407] Connection header received
0924 23:10:36.792946 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.792949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.792973 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.793060 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.793769 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.794656 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.795418 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.795780 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.795892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.798309 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.798317 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.798317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.798329 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.800071 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.800447 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.800450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.800450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.800509 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.800716 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.800717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.800717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.800951 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.801098 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.801497 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.801575 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260924 23:10:36.802014 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:36.809127 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.802369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.0.98.61:39435)
0924 23:10:36.802436 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.802438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.802446 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:36.802472 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.802473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.802500 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.802557 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.802858 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.803546 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.804425 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.804640 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.804693 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.806636 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.806638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.806638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.806646 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.808045 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.808388 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.808389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.808390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.808445 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.808632 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.808633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.808634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.808744 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.808814 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.808959 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.809027 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:10:36.809127 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.802306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39435 (local address 127.0.0.1:37316)
0924 23:10:36.802346 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.802356 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.802391 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.802578 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.802578 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.802748 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.802786 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.802788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.803612 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.803613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.804205 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.804208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.804712 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.804713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.804783 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.805184 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.805189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.806552 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.808135 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.808136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.808136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.808300 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.808508 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.808508 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.808509 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.808564 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.808807 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.808808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.808852 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.808934 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.809026 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 23:10:36.809497 22799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:36.811203 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.803626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.0.98.62:44437)
0924 23:10:36.803683 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.803685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.803692 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:10:36.803710 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.803711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.803720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.803778 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.804212 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.804705 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.805334 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.805629 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.805704 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.808268 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.808269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.808269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.808277 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.810072 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.810417 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.810418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.810418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.810470 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.810686 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.810687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.810687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.810824 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.810995 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.811032 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.811104 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:10:36.815938 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.809763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58114)
0924 23:10:36.809801 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.809810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.809842 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.810008 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.810008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.810111 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.810138 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.810139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.810825 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.810826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.811392 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.811394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.811764 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.811765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.811835 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.812126 (+ 291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.812130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.813261 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.815092 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.815093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.815093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.815245 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.815443 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.815444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.815444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.815491 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.815729 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.815730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.815761 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.815809 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.815880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:10:36.816044 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.809832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.0.98.60:34499)
0924 23:10:36.809874 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.809895 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.809903 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:36.809923 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.809923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.809934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.810006 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.810208 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.810761 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.811450 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.811661 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.811732 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.813350 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.813351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.813351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.813357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.815011 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.815318 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.815319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.815320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.815376 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.815539 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.815540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.815540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.815667 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.815736 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.815840 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.815931 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":25}
I20260924 23:10:36.817765 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: Bootstrap starting.
I20260924 23:10:36.818614 22799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:36.818956 22799 log.cc:824] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:36.818975 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.811780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.0.98.61:39435)
0924 23:10:36.811832 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.811841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.811850 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:36.811872 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.811873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.811883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.811948 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.812572 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.813039 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.813764 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.814044 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.814099 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.816511 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.816513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.816513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.816520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.817832 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.818216 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.818218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.818218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.818270 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.818474 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.818475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.818476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.818588 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.818764 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.818809 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.818881 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 23:10:36.819818 22799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: No bootstrap required, opened a new log
I20260924 23:10:36.820273 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:36.820557 22649 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } attempt: 2
I20260924 23:10:36.822120 22799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.822263 22799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501a5784ed244207831cd94ce3be793c, State: Initialized, Role: FOLLOWER
I20260924 23:10:36.821431 22623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "99436a9dee524bb884134ce6e8e06271"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40122
I20260924 23:10:36.822371 22799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.822634 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.822700 22799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:36.822721 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 23:10:36.824440 22816 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:36.824494 22816 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:10:36.824945 22731 master_runner.cc:186] Error getting master registration for 127.0.98.62:44437: OK, instance_id { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" instance_seqno: 1790291436478454 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:10:36.827658 22731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:10:36.828470 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.820798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58120)
0924 23:10:36.820852 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.820868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.820907 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.821235 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.821236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.821410 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.821452 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.821454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.822359 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.822360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.823149 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.823152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.823608 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.823610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.823683 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.824119 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.824124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.825528 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.827308 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.827309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.827310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.827528 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.827752 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.827753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.827753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.827815 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.828144 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.828145 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.828192 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.828274 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.828377 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:10:36.828482 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.821011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.60:34499)
0924 23:10:36.821128 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.821130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.821136 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:10:36.821152 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.821152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.821178 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.821234 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.821532 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.822260 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.823241 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.823533 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.823591 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.825615 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.825617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.825617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.825624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.827226 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.827612 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.827613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.827613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.827670 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.827870 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.827871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.827871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.828069 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.828139 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.828317 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.828391 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 23:10:36.829769 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271: Bootstrap starting.
I20260924 23:10:36.830406 22649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:36.830688 22649 log.cc:824] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:36.831411 22649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271: No bootstrap required, opened a new log
I20260924 23:10:36.832445 22649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.832569 22649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99436a9dee524bb884134ce6e8e06271, State: Initialized, Role: FOLLOWER
I20260924 23:10:36.832660 22649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.833016 22649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:36.832978 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.833066 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:36.833155 22623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 23:10:36.833226 22623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:36.834151 22623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.834410 22623 leader_election.cc:288] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [CANDIDATE]: Term 1 election: Requested vote from peers 22d6be21866a4f1080497f76f50de1f3 (127.0.98.61:39435), 501a5784ed244207831cd94ce3be793c (127.0.98.60:34499)
I20260924 23:10:36.834733 22773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99436a9dee524bb884134ce6e8e06271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "501a5784ed244207831cd94ce3be793c"
I20260924 23:10:36.834838 22773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:36.836187 22773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99436a9dee524bb884134ce6e8e06271 in term 1.
W20260924 23:10:36.835431 22830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 23:10:36.836416 22600 leader_election.cc:302] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501a5784ed244207831cd94ce3be793c, 99436a9dee524bb884134ce6e8e06271; no voters:
I20260924 23:10:36.836880 22819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:36.837001 22819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [term 1 LEADER]: Becoming Leader. State: Replica: 99436a9dee524bb884134ce6e8e06271, State: Running, Role: LEADER
I20260924 23:10:36.837071 22819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.837451 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99436a9dee524bb884134ce6e8e06271. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.837527 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:36.838536 22581 master_runner.cc:186] Error getting master registration for 127.0.98.61:39435: OK, instance_id { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" instance_seqno: 1790291436623035 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:10:36.840797 22581 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 23:10:36.840864 22830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:36.841445 22838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:36.841866 22838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:36.843578 22773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [term 1 FOLLOWER]: Refusing update from remote peer 99436a9dee524bb884134ce6e8e06271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:36.844242 22832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:36.845382 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99436a9dee524bb884134ce6e8e06271. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.845467 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:36.847316 22842 mvcc.cc:205] Tried to move back new op lower bound from 7333033725308846080 to 7333033725285314560. Current Snapshot: MvccSnapshot[applied={T|T < 7333033725308846080}]
I20260924 23:10:36.847570 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.847657 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:36.848031 22805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.848106 22805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:36.848328 22843 mvcc.cc:205] Tried to move back new op lower bound from 7333033725308846080 to 7333033725285314560. Current Snapshot: MvccSnapshot[applied={T|T < 7333033725308846080}]
I20260924 23:10:36.849146 22838 catalog_manager.cc:1347] Generated new cluster ID: c549526fbcf24fa48aa25202cc02ee8f
I20260924 23:10:36.849200 22838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:36.857867 22724 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } attempt: 3
I20260924 23:10:36.859973 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.837775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.0.98.62:44437)
0924 23:10:36.837814 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.837816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.842665 (+ 4849us) server_negotiation.cc:407] Connection header received
0924 23:10:36.842759 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.842760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.842769 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.842797 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.844198 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.844926 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.846549 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.846854 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.846923 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.851319 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.851321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.851321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.851328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.853106 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.855119 (+ 2013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.855120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.855121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.855195 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.858686 (+ 3491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.858687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.858688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.858848 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.858931 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.859800 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.859880 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:36.863137 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.838145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.0.98.60:34499)
0924 23:10:36.838183 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.838185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.846017 (+ 7832us) server_negotiation.cc:407] Connection header received
0924 23:10:36.846081 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.846082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.846092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.846119 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.846844 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.847478 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.848494 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.848758 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.848802 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.856695 (+ 7893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.856696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.856697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.856703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.858342 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.861511 (+ 3169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.861512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.861513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.861586 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.862155 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.862156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.862156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.862302 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.862383 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.862965 (+ 582us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.863046 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 23:10:36 dist-test-slave-ph9j krb5kdc[22568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:10:36.871239 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.838489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.0.98.61:39435)
0924 23:10:36.838527 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.838529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.849199 (+ 10670us) server_negotiation.cc:407] Connection header received
0924 23:10:36.849299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.849300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.849311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.849340 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.855445 (+ 6105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.856175 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.864463 (+ 8288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.865349 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.865406 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.867978 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.867980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.867980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.867987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.869399 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.870094 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.870109 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.870109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.870170 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.870475 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.870476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.870477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.870622 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.870745 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.871067 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.871145 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:36.871477 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.858114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34499 (local address 127.0.0.1:58128)
0924 23:10:36.858185 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:36.858194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:36.858223 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:36.863310 (+ 5087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:36.863312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:36.863472 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:36.863515 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.863517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.864339 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.864340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.865119 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:36.865122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.865548 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.865549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:36.865594 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.866195 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:36.866200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:36.867508 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:36.870517 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.870518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.870519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.870699 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.870946 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:36.870947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:36.870947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:36.870995 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:36.871213 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:36.871214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:36.871251 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 23:10:36.871307 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 23:10:36.871390 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 23:10:36.871773 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:36.858282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.0.98.60:34499)
0924 23:10:36.863201 (+ 4919us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.863203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.863210 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:10:36.863231 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.863232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.863242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.863302 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.863600 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.864256 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.865195 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:36.865464 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:36.865509 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:36.867588 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:36.867589 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:36.867589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:36.867594 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:36.870436 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.870828 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.870829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.870829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.870885 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:36.871046 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:36.871046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:36.871047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:36.871151 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:36.871194 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:36.871627 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 23:10:36.871688 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4906,"thread_start_us":75,"threads_started":1}
I20260924 23:10:36.873327 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: Bootstrap starting.
I20260924 23:10:36.874135 22724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:36.874449 22724 log.cc:824] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:36.875463 22724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: No bootstrap required, opened a new log
I20260924 23:10:36.879907 22724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.880257 22724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d6be21866a4f1080497f76f50de1f3, State: Initialized, Role: FOLLOWER
I20260924 23:10:36.880474 22724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } }
I20260924 23:10:36.881181 22698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "99436a9dee524bb884134ce6e8e06271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "22d6be21866a4f1080497f76f50de1f3"
I20260924 23:10:36.881289 22698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:36.882207 22698 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99436a9dee524bb884134ce6e8e06271 in term 1.
I20260924 23:10:36.882652 22697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [term 1 FOLLOWER]: Refusing update from remote peer 99436a9dee524bb884134ce6e8e06271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:36.882961 22846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:36.884512 22724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 23:10:36.889178 22865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:36.889279 22865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:36.890072 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.890165 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:36.890244 22838 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:36.890921 22838 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:36.890887 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.891059 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:36.891153 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99436a9dee524bb884134ce6e8e06271. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.891250 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:36.891317 22854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99436a9dee524bb884134ce6e8e06271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99436a9dee524bb884134ce6e8e06271" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44437 } } peers { permanent_uuid: "22d6be21866a4f1080497f76f50de1f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39435 } } peers { permanent_uuid: "501a5784ed244207831cd94ce3be793c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34499 } } }
I20260924 23:10:36.891404 22854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:36.891949 22852 mvcc.cc:205] Tried to move back new op lower bound from 7333033725308846080 to 7333033725285314560. Current Snapshot: MvccSnapshot[applied={T|T < 7333033725308846080}]
I20260924 23:10:36.896377 22838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99436a9dee524bb884134ce6e8e06271: Generated new TSK 0
I20260924 23:10:36.896534 22838 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 23:10:36.926924 22653 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.925632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.0.98.62:44437)
0924 23:10:36.925675 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.925677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.926274 (+ 597us) server_negotiation.cc:407] Connection header received
0924 23:10:36.926378 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.926379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.926390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.926421 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.926856 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40142: BlockingRecv error: recv got EOF from 127.0.0.1:40142 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:10:36.927873 22848 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.925994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58136 (local address 127.0.98.60:34499)
0924 23:10:36.926052 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.926054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.927279 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 23:10:36.927438 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.927439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.927450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.927490 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.927810 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58136: BlockingRecv error: recv got EOF from 127.0.0.1:58136 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:10:36.928941 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.928271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.61:39435)
0924 23:10:36.928318 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.928319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.928422 (+ 103us) server_negotiation.cc:407] Connection header received
0924 23:10:36.928504 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.928505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.928514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.928541 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.928883 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37332: BlockingRecv error: recv got EOF from 127.0.0.1:37332 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 23:10:36.931232 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.930537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.0.98.61:39435)
0924 23:10:36.930570 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.930572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.930783 (+ 211us) server_negotiation.cc:407] Connection header received
0924 23:10:36.930878 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.930879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.930895 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.930922 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.931192 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37338: BlockingRecv error: recv got EOF from 127.0.0.1:37338 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:10:36.934345 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.933552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.61:39435)
0924 23:10:36.933583 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.933584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.933858 (+ 274us) server_negotiation.cc:407] Connection header received
0924 23:10:36.933974 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.933975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.933984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.934011 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.934303 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37348: BlockingRecv error: recv got EOF from 127.0.0.1:37348 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:10:36.938237 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.937488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.0.98.61:39435)
0924 23:10:36.937520 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.937522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.937735 (+ 213us) server_negotiation.cc:407] Connection header received
0924 23:10:36.937831 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.937832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.937842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.937870 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.938198 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37356: BlockingRecv error: recv got EOF from 127.0.0.1:37356 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 23:10:36.944146 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.943424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.0.98.61:39435)
0924 23:10:36.943456 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.943458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.943654 (+ 196us) server_negotiation.cc:407] Connection header received
0924 23:10:36.943762 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.943763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.943773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.943800 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.944105 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37372: BlockingRecv error: recv got EOF from 127.0.0.1:37372 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 23:10:36.954244 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.953394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.0.98.61:39435)
0924 23:10:36.953429 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.953431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.953671 (+ 240us) server_negotiation.cc:407] Connection header received
0924 23:10:36.953786 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.953787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.953797 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.953826 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.954185 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37376: BlockingRecv error: recv got EOF from 127.0.0.1:37376 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 23:10:36.972559 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:36.971667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.0.98.61:39435)
0924 23:10:36.971708 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:36.971726 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:36.971972 (+ 246us) server_negotiation.cc:407] Connection header received
0924 23:10:36.972075 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:36.972076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:36.972088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:36.972121 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:36.972497 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37382: BlockingRecv error: recv got EOF from 127.0.0.1:37382 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 23:10:37.006989 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:37.006103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.0.98.61:39435)
0924 23:10:37.006148 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.006150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.006408 (+ 258us) server_negotiation.cc:407] Connection header received
0924 23:10:37.006504 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.006505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.006517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.006546 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:37.006931 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37396: BlockingRecv error: recv got EOF from 127.0.0.1:37396 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:10:37.073534 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:37.072637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.98.61:39435)
0924 23:10:37.072717 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.072719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.072914 (+ 195us) server_negotiation.cc:407] Connection header received
0924 23:10:37.072949 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.072950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.072962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.073025 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:37.073456 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37412: BlockingRecv error: recv got EOF from 127.0.0.1:37412 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 23:10:37.203428 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:37.202687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.0.98.61:39435)
0924 23:10:37.202754 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.202756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.202885 (+ 129us) server_negotiation.cc:407] Connection header received
0924 23:10:37.202934 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.202935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.202946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.202998 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:37.203370 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37424: BlockingRecv error: recv got EOF from 127.0.0.1:37424 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 23:10:37.461714 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:37.460913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.0.98.61:39435)
0924 23:10:37.460986 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.460988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.461073 (+ 85us) server_negotiation.cc:407] Connection header received
0924 23:10:37.461160 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.461161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.461175 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.461248 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:37.461652 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50762: BlockingRecv error: recv got EOF from 127.0.0.1:50762 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 23:10:37.824928 22816 catalog_manager.cc:1259] Loaded cluster ID: c549526fbcf24fa48aa25202cc02ee8f
I20260924 23:10:37.825004 22816 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: loading cluster ID for follower catalog manager: success
I20260924 23:10:37.825839 22816 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: acquiring CA information for follower catalog manager: success
I20260924 23:10:37.826174 22816 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 501a5784ed244207831cd94ce3be793c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:10:37.889791 22865 catalog_manager.cc:1259] Loaded cluster ID: c549526fbcf24fa48aa25202cc02ee8f
I20260924 23:10:37.889896 22865 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: loading cluster ID for follower catalog manager: success
I20260924 23:10:37.890664 22865 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: acquiring CA information for follower catalog manager: success
I20260924 23:10:37.890947 22865 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22d6be21866a4f1080497f76f50de1f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:10:37.977319 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:37.975013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.0.98.61:39435)
0924 23:10:37.975111 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.975113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.975122 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:37.975166 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.975166 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.975212 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.975306 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:37.975451 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.975931 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.976646 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.976846 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.976900 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:37.977040 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:37.977143 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:37.977199 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:37.977237 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 23:10:37.977254 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260924 23:10:37.981323 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:37.979129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.98.60:34499)
0924 23:10:37.979221 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.979223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.979238 (+ 15us) server_negotiation.cc:407] Connection header received
0924 23:10:37.979295 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.979296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.979329 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.979393 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:37.979480 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.979940 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.980619 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.980812 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.980873 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:37.981006 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:37.981099 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:37.981157 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:37.981196 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 23:10:37.981214 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":51,"threads_started":1}
I20260924 23:10:37.986056 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:37.983002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.0.98.61:39435)
0924 23:10:37.983064 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.983066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.983142 (+ 76us) server_negotiation.cc:407] Connection header received
0924 23:10:37.983250 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.983251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.983265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.983297 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:37.983482 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.984201 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.985240 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.985514 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.985551 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:37.985745 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:37.985776 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:37.985805 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:37.985944 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 23:10:37.985968 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 23:10:37.986086 22939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:37.983283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.0.98.60:34499)
0924 23:10:37.983318 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.983319 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.983326 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:10:37.983344 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.983345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.983353 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.983419 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:37.983538 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.984064 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.985254 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.985536 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.985617 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:37.985784 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:37.985816 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:37.985866 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:37.985963 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 23:10:37.985987 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:37.986466 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:37.983042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.0.98.62:44437)
0924 23:10:37.983277 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:37.983279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:37.983286 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:10:37.983332 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:37.983333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:37.983385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:37.983600 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:37.983640 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.984452 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.985323 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:37.985631 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:37.985713 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:37.985907 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:37.986009 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:37.986069 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:37.986355 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 23:10:37.986382 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":61,"threads_started":1}
I20260924 23:10:37.988024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22581
I20260924 23:10:37.993484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22656
I20260924 23:10:37.998307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22731
2026-09-24T23:10:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2785 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 23:10:38 dist-test-slave-ph9j krb5kdc[22950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 23:10:38 dist-test-slave-ph9j krb5kdc[22950](info): set up 2 sockets
Sep 24 23:10:38 dist-test-slave-ph9j krb5kdc[22950](info): commencing operation
krb5kdc: starting...
W20260924 23:10:39.034194 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/test-user.keytab.
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T23:10:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T23:10:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 23:10:39.157768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39733
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39733,127.0.98.61:39945,127.0.98.60:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 23:10:39.239516 22963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:10:39.239730 22963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:39.239748 22963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:39.239776 22963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:39.240927 22963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:39.240967 22963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:39.240981 22963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:39.240994 22963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:39.241006 22963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:39.242601 22963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39733,127.0.98.61:39945,127.0.98.60:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:39.242859 22963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:39.243086 22963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:39.247907 22968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:39.247866 22969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:39.247880 22971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:39.248034 22963 server_base.cc:1034] running on GCE node
I20260924 23:10:39.248353 22963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:39.248637 22963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:39.249794 22963 hybrid_clock.cc:648] HybridClock initialized: now 1790291439249781 us; error 47 us; skew 500 ppm
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:39.252086 22963 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 23:10:39.252503 22963 webserver.cc:460] Webserver started at http://127.0.98.62:36105/ using document root <none> and password file <none>
I20260924 23:10:39.252731 22963 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:39.252800 22963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:39.252948 22963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:39.254088 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data/instance:
uuid: "d3c8e6c101e54d929f23c32730df75f1"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "fae008177290ab9d8a8078e05074e2d3"
server_key_iv: "648f44dd00f482544adf4e7069a29598"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.254442 22963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3c8e6c101e54d929f23c32730df75f1"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "fae008177290ab9d8a8078e05074e2d3"
server_key_iv: "648f44dd00f482544adf4e7069a29598"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.255693 22963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 23:10:39.256386 22978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:39.256544 22963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:39.256645 22963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal
uuid: "d3c8e6c101e54d929f23c32730df75f1"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "fae008177290ab9d8a8078e05074e2d3"
server_key_iv: "648f44dd00f482544adf4e7069a29598"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.256716 22963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:39.270239 22963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:39.271957 22963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:39.272118 22963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:39.276038 22963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39733
I20260924 23:10:39.276073 23030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39733 every 8 connection(s)
I20260924 23:10:39.276410 22963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/data/info.pb
I20260924 23:10:39.277393 23031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:39.279047 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:39.282480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 22963
I20260924 23:10:39.282577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-0/wal/instance
I20260924 23:10:39.282765 392 external_mini_cluster.cc:1428] Setting key d0ca223d58ba81b7a0aa52ca7a5ec8f9
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:39.288916 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.279702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39733 (local address 127.0.0.1:50662)
0924 23:10:39.279886 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.279927 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.280435 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.280729 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.280731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.280996 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.281146 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.281150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.282224 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.282224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.283323 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.283326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.283833 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.283834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.283926 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.284504 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.284512 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.285790 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.287496 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.287502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.287503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.287693 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.287930 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.287930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.287931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.288016 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.288356 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.288360 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.288419 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.288527 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.288742 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":81,"thread_start_us":82,"threads_started":1}
I20260924 23:10:39.288923 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.279745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50662 (local address 127.0.98.62:39733)
0924 23:10:39.279895 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.279905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.280021 (+ 116us) server_negotiation.cc:407] Connection header received
0924 23:10:39.280588 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.280591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.280610 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.280723 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.281251 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.282122 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.283411 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.283716 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.283806 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.285911 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.285918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.285918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.285925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.287401 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.287787 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.287789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.287790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.287864 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.288092 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.288093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.288093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.288290 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.288376 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.288550 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.288689 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260924 23:10:39.289706 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 23:10:39.290087 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.289971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48176)
0924 23:10:39.290016 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.290057 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:10:39.290198 22981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
I20260924 23:10:39.290585 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.290492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48190)
0924 23:10:39.290539 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.290563 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 23:10:39.290695 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 23:10:39.308466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39945
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39733,127.0.98.61:39945,127.0.98.60:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:10:39.339939 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } attempt: 1
I20260924 23:10:39.340535 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.340391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48204)
0924 23:10:39.340459 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.340505 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 23:10:39.340942 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.340864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48214)
0924 23:10:39.340900 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.340926 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 23:10:39.341043 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
W20260924 23:10:39.390641 23038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:10:39.390882 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:39.390911 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:39.390967 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:39.392532 23038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:39.392592 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:39.392622 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:39.392644 23038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:39.392674 23038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:39.394480 23038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39733,127.0.98.61:39945,127.0.98.60:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:39.394805 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:39.395049 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:39.400514 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:39.400584 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:39.400518 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:39.400856 23038 server_base.cc:1034] running on GCE node
I20260924 23:10:39.401113 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:39.401427 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:39.402572 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790291439402561 us; error 41 us; skew 500 ppm
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:39.405014 23038 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 23:10:39.405508 23038 webserver.cc:460] Webserver started at http://127.0.98.61:40679/ using document root <none> and password file <none>
I20260924 23:10:39.405755 23038 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:39.405823 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:39.405999 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:39.407055 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data/instance:
uuid: "8b113a23d5ef4496a2db5de4183a6f8b"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "cbc9e8f2e3b8d5d3f56620718ffae86d"
server_key_iv: "863a85e84ddfc4052b7445b4cb8cf33a"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.407424 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal/instance:
uuid: "8b113a23d5ef4496a2db5de4183a6f8b"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "cbc9e8f2e3b8d5d3f56620718ffae86d"
server_key_iv: "863a85e84ddfc4052b7445b4cb8cf33a"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.408739 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:39.409181 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } attempt: 2
I20260924 23:10:39.409490 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:39.409571 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.409432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48218)
0924 23:10:39.409487 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.409523 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 23:10:39.409646 23038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:39.409749 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal
uuid: "8b113a23d5ef4496a2db5de4183a6f8b"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "cbc9e8f2e3b8d5d3f56620718ffae86d"
server_key_iv: "863a85e84ddfc4052b7445b4cb8cf33a"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.409823 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:39.410025 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.409935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48234)
0924 23:10:39.409979 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.410002 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 23:10:39.410169 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39945: connect: Connection refused (error 111)
I20260924 23:10:39.427656 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:39.429486 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:39.429656 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:39.433687 23038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39945
I20260924 23:10:39.433730 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39945 every 8 connection(s)
I20260924 23:10:39.434124 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/data/info.pb
I20260924 23:10:39.435135 23106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:39.436671 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:39.443589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 23038
I20260924 23:10:39.443686 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-1/wal/instance
I20260924 23:10:39.443926 392 external_mini_cluster.cc:1428] Setting key e1e3c2d8c992fff9df4c0a5ba5d0c247
I20260924 23:10:39.445194 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.437356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50664 (local address 127.0.98.62:39733)
0924 23:10:39.437432 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.437434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.437631 (+ 197us) server_negotiation.cc:407] Connection header received
0924 23:10:39.438083 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.438084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.438094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.438155 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.438692 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.439158 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.440112 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.440313 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.440368 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.442453 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.442455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.442455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.442460 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.443729 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.444161 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.444162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.444163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.444241 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.444512 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.444513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.444514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.444662 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.444730 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.444963 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.445052 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 23:10:39.445205 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.437341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39733 (local address 127.0.0.1:50664)
0924 23:10:39.437500 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.437542 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.438005 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.438206 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.438209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.438438 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.438626 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.438629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.439231 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.439231 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.440029 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.440031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.440394 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.440395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.440494 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.441022 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.441030 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.442377 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.443810 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.443817 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.443819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.444090 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.444320 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.444320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.444321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.444446 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.444727 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.444731 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.444840 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.444920 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.445032 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 23:10:39.446022 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 23:10:39.453755 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.446298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48246)
0924 23:10:39.446348 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.446360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.446399 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.446651 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.446651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.446800 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.446841 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.446842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.447810 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.447811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.448308 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.448309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.448833 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.448834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.448901 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.449380 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.449384 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.450883 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.452609 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.452610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.452610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.452775 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.452989 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.452990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.452990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.453043 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.453330 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.453331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.453368 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.453439 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.453595 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 23:10:39.453766 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.446372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.0.98.61:39945)
0924 23:10:39.446527 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.446531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.446544 (+ 13us) server_negotiation.cc:407] Connection header received
0924 23:10:39.446560 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.446562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.446571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.446650 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.446921 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.447724 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.448401 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.448735 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.448833 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.450983 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.450990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.450991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.450997 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.452533 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.452865 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.452868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.452869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.452924 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.453113 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.453114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.453114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.453260 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.453351 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.453469 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.453650 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 23:10:39.454496 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:39.454839 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.454718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:45980)
0924 23:10:39.454765 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.454809 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 23:10:39.454996 23056 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
I20260924 23:10:39.455382 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.455299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:45988)
0924 23:10:39.455344 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.455367 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 23:10:39.455499 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38367: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 23:10:39.469282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKo6W1P/build/release/bin/kudu
/tmp/dist-test-taskKo6W1P/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38367
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39733,127.0.98.61:39945,127.0.98.60:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 23:10:39.501322 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } attempt: 3
I20260924 23:10:39.504732 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } attempt: 1
I20260924 23:10:39.505172 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.505028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46000)
0924 23:10:39.505095 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.505133 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 23:10:39.505523 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.505417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46002)
0924 23:10:39.505471 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.505499 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 23:10:39.505615 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38367: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:10:39.508466 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.501780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.98.61:39945)
0924 23:10:39.501851 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.501854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.501866 (+ 12us) server_negotiation.cc:407] Connection header received
0924 23:10:39.501927 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.501928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.501948 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.502010 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.502338 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.502905 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.503607 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.503821 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.503898 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.506066 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.506067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.506068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.506074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.507440 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.507816 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.507817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.507818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.507866 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.508057 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.508057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.508058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.508154 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.508204 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.508360 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.508416 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 23:10:39.508527 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.501688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48254)
0924 23:10:39.501750 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.501763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.501838 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.502015 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.502016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.502187 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.502253 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.502255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.502990 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.502992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.503519 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.503521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.503912 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.503913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.503992 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.504509 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.504514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.505991 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.507522 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.507523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.507523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.507731 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.507932 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.507933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.507934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.507998 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.508220 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.508221 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.508271 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.508345 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.508435 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 23:10:39.509032 23031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
I20260924 23:10:39.509373 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.509268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46008)
0924 23:10:39.509315 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.509349 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 23:10:39.509732 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.509652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46014)
0924 23:10:39.509691 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.509716 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 23:10:39.509826 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38367: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
I20260924 23:10:39.540990 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } attempt: 1
I20260924 23:10:39.541505 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.541349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46026)
0924 23:10:39.541421 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.541468 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 23:10:39.541848 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.541769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46042)
0924 23:10:39.541796 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.541826 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 23:10:39.541960 23031 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38367: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
W20260924 23:10:39.550954 23113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 23:10:39.551146 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 23:10:39.551163 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 23:10:39.551182 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 23:10:39.552218 23113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 23:10:39.552253 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 23:10:39.552268 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 23:10:39.552280 23113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 23:10:39.552292 23113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 23:10:39.553682 23113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39733,127.0.98.61:39945,127.0.98.60:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 22:10:57 UTC on ccycloud.quasar-rfzvvf.root.comops.site
build id 4671452
I20260924 23:10:39.553936 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 23:10:39.554131 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 23:10:39.558658 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:39.558668 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 23:10:39.558671 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 23:10:39.558738 23113 server_base.cc:1034] running on GCE node
I20260924 23:10:39.559104 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 23:10:39.559459 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 23:10:39.560628 23113 hybrid_clock.cc:648] HybridClock initialized: now 1790291439560609 us; error 37 us; skew 500 ppm
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 23:10:39.563171 23113 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 23:10:39.563612 23113 webserver.cc:460] Webserver started at http://127.0.98.60:41113/ using document root <none> and password file <none>
I20260924 23:10:39.563848 23113 fs_manager.cc:359] Metadata directory not provided
I20260924 23:10:39.563916 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 23:10:39.564077 23113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 23:10:39.565110 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data/instance:
uuid: "5e4017755acc4aeab2a772af86a02f97"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "97c5592191309198ff8c2f04004e0d16"
server_key_iv: "4750289080fbe3db4a47c5a11e06e526"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.565474 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal/instance:
uuid: "5e4017755acc4aeab2a772af86a02f97"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "97c5592191309198ff8c2f04004e0d16"
server_key_iv: "4750289080fbe3db4a47c5a11e06e526"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.566866 23113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 23:10:39.567608 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 23:10:39.567783 23113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 23:10:39.567888 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal
uuid: "5e4017755acc4aeab2a772af86a02f97"
format_stamp: "Formatted at 2026-09-24 23:10:39 on dist-test-slave-ph9j"
server_key: "97c5592191309198ff8c2f04004e0d16"
server_key_iv: "4750289080fbe3db4a47c5a11e06e526"
server_key_version: "encryptionkey@0"
I20260924 23:10:39.567958 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 23:10:39.573760 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } attempt: 2
I20260924 23:10:39.574175 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.574047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46056)
0924 23:10:39.574108 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.574146 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 23:10:39.574504 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.574425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46072)
0924 23:10:39.574464 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.574487 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 23:10:39.574618 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38367: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38367: connect: Connection refused (error 111)
I20260924 23:10:39.577387 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 23:10:39.579226 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 23:10:39.579385 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 23:10:39.583554 23113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38367
I20260924 23:10:39.583580 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38367 every 8 connection(s)
I20260924 23:10:39.584084 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/data/info.pb
I20260924 23:10:39.585057 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 23:10:39.586649 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:39.594062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKo6W1P/build/release/bin/kudu as pid 23113
I20260924 23:10:39.594184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKo6W1P/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790291169445806-392-0/minicluster-data/master-2/wal/instance
I20260924 23:10:39.594412 392 external_mini_cluster.cc:1428] Setting key bdef730bbb1abbb2d5a6052e2a64273c
I20260924 23:10:39.595660 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.587415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.0.98.62:39733)
0924 23:10:39.587491 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.587493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.587757 (+ 264us) server_negotiation.cc:407] Connection header received
0924 23:10:39.588177 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.588178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.588188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.588244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.588848 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.589332 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.590434 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.590619 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.590679 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.592748 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.592749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.592749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.592755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.594095 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.594658 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.594659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.594660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.594720 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.595015 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.595016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.595017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.595135 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.595213 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.595491 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.595568 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 23:10:39.595884 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.587406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39733 (local address 127.0.0.1:50678)
0924 23:10:39.587620 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.587655 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.588090 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.588323 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.588327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.588571 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.588771 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.588774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.589417 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.589418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.590358 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.590360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.590704 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.590705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.590820 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.591343 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.591351 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.592671 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.594265 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.594273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.594275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.594581 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.594787 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.594788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.594788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.594934 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.595206 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.595226 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.595358 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.595470 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.595625 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":126,"threads_started":1}
I20260924 23:10:39.596379 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:10:39.603368 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.596711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.0.98.61:39945)
0924 23:10:39.596754 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.596756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.596791 (+ 35us) server_negotiation.cc:407] Connection header received
0924 23:10:39.596864 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.596865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.596874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.596929 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.597194 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.597641 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.598368 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.598581 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.598662 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.600832 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.600834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.600834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.600841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.602303 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.602608 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.602609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.602609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.602661 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.602868 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.602869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.602870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.602980 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.603052 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.603197 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.603272 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:39.603368 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.596660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39945 (local address 127.0.0.1:48258)
0924 23:10:39.596749 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.596760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.596805 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.596933 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.596933 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.597063 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.597107 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.597108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.597724 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.597725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.598287 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.598289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.598662 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.598663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.598726 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.599111 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.599114 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.600749 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.602369 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.602370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.602370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.602528 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.602742 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.602743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.602744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.602798 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.603043 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.603044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.603085 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.603172 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.603281 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 23:10:39.603679 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.594833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.98.60:38367)
0924 23:10:39.595020 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.595025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.595051 (+ 26us) server_negotiation.cc:407] Connection header received
0924 23:10:39.595100 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.595102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.595116 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.595180 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.595845 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.596757 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.597511 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.597826 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.597923 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.600301 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.600309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.600310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.600317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.602228 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.602570 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.602573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.602574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.602628 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.602836 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.602837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.602837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.602974 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.603116 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.603202 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.603369 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260924 23:10:39.603732 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 23:10:39.610220 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.603998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46090)
0924 23:10:39.604048 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.604059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.604093 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.604377 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.604378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.604524 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.604557 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.604564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.605177 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.605178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.605630 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.605631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.606090 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.606091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.606139 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.606483 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.606488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.607705 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.609172 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.609173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.609174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.609331 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.609529 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.609530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.609531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.609582 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.609836 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.609837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.609873 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.609978 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.610068 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 23:10:39.610265 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.604254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.0.98.60:38367)
0924 23:10:39.604290 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.604292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.604298 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:10:39.604315 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.604316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.604328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.604368 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.604641 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.605098 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.605712 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.606012 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.606090 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.607786 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.607788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.607788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.607794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.609068 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.609412 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.609413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.609413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.609460 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.609664 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.609665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.609665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.609764 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.609833 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.610001 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.610113 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":19}
I20260924 23:10:39.611479 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: Bootstrap starting.
I20260924 23:10:39.611616 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.604544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.0.98.62:39733)
0924 23:10:39.604581 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.604583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.604590 (+ 7us) server_negotiation.cc:407] Connection header received
0924 23:10:39.604615 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.604616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.604625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.604678 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.605125 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.605574 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.606324 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.606592 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.606671 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.608972 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.608973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.608974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.608981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.610532 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.610864 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.610865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.610866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.610922 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.611110 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.611111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.611111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.611224 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.611408 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.611443 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.611512 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:39.612535 23181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:39.612900 23181 log.cc:824] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:39.613708 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: No bootstrap required, opened a new log
I20260924 23:10:39.615528 23181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.615648 23181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4017755acc4aeab2a772af86a02f97, State: Initialized, Role: FOLLOWER
I20260924 23:10:39.615744 23181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.616078 23181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:39.616058 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.616140 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 23:10:39.617107 23031 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } attempt: 2
W20260924 23:10:39.618181 23198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:39.618234 23198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:39.620772 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.612275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.98.61:39945)
0924 23:10:39.612316 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.612318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.612326 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:39.612348 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.612349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.612358 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.612415 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.612873 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.613546 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.614337 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.614613 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.614682 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.616841 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.616842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.616842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.616849 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.619506 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.619938 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.619939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.619940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.620002 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.620217 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.620218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.620219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.620351 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.620524 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.620578 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.620656 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:10:39.618471 23113 master_runner.cc:186] Error getting master registration for 127.0.98.62:39733: OK, instance_id { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" instance_seqno: 1790291439274915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:10:39.621459 23113 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:10:39.622784 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 23:10:39.624303 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.617375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46098)
0924 23:10:39.617420 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.617432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.617473 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.617689 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.617690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.617848 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.617929 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.617931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.618735 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.618736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.619278 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.619280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.619649 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.619650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.619706 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.620098 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.620103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.621390 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.623160 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.623161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.623161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.623390 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.623588 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.623589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.623589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.623657 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.623979 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.623980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.624029 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.624105 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.624206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 23:10:39.624321 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.617477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.0.98.60:38367)
0924 23:10:39.617557 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.617559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.617567 (+ 8us) server_negotiation.cc:407] Connection header received
0924 23:10:39.617587 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.617588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.617596 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.617673 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.617998 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.618647 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.619363 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.619565 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.619647 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.621474 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.621475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.621476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.621482 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.623073 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.623477 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.623478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.623479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.623532 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.623727 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.623728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.623728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.623894 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.623967 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.624143 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.624234 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 23:10:39.623849 23005 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d3c8e6c101e54d929f23c32730df75f1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50688
I20260924 23:10:39.626224 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1: Bootstrap starting.
I20260924 23:10:39.626823 23031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:39.627103 23031 log.cc:824] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:39.627821 23031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1: No bootstrap required, opened a new log
I20260924 23:10:39.628769 23031 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.628875 23031 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3c8e6c101e54d929f23c32730df75f1, State: Initialized, Role: FOLLOWER
I20260924 23:10:39.628968 23031 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.629290 23031 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:39.629336 23005 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 23:10:39.629402 23005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 23:10:39.630276 23005 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.630496 23005 leader_election.cc:288] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [CANDIDATE]: Term 1 election: Requested vote from peers 8b113a23d5ef4496a2db5de4183a6f8b (127.0.98.61:39945), 5e4017755acc4aeab2a772af86a02f97 (127.0.98.60:38367)
I20260924 23:10:39.630908 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3c8e6c101e54d929f23c32730df75f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e4017755acc4aeab2a772af86a02f97"
I20260924 23:10:39.631017 23155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [term 0 FOLLOWER]: Advancing to term 1
W20260924 23:10:39.631376 23212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 23:10:39.631479 23212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 23:10:39.632160 23155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c8e6c101e54d929f23c32730df75f1 in term 1.
I20260924 23:10:39.632683 22981 leader_election.cc:302] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4017755acc4aeab2a772af86a02f97, d3c8e6c101e54d929f23c32730df75f1; no voters:
I20260924 23:10:39.629303 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.633826 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:39.633950 23201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 23:10:39.634039 23201 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [term 1 LEADER]: Becoming Leader. State: Replica: d3c8e6c101e54d929f23c32730df75f1, State: Running, Role: LEADER
I20260924 23:10:39.634177 23201 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.631759 22963 master_runner.cc:186] Error getting master registration for 127.0.98.61:39945: OK, instance_id { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" instance_seqno: 1790291439432497 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 23:10:39.634724 22963 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 23:10:39.634639 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c8e6c101e54d929f23c32730df75f1. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.634835 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:39.635051 23220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 23:10:39.635246 23220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 23:10:39.636873 23155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [term 1 FOLLOWER]: Refusing update from remote peer d3c8e6c101e54d929f23c32730df75f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:39.637182 23201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:39.638428 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c8e6c101e54d929f23c32730df75f1. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.638542 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:39.643836 23214 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.643954 23214 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:39.644850 23187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.644955 23187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:39.645246 23222 mvcc.cc:205] Tried to move back new op lower bound from 7333033736750100480 to 7333033736742621184. Current Snapshot: MvccSnapshot[applied={T|T < 7333033736750100480}]
I20260924 23:10:39.652256 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.642699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.0.98.60:38367)
0924 23:10:39.642737 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.642739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.643749 (+ 1010us) server_negotiation.cc:407] Connection header received
0924 23:10:39.643849 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.643850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.643861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.643891 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.644678 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.645429 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.646632 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.646925 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.646964 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.649414 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.649416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.649416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.649423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.651046 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.651439 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.651440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.651441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.651490 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.651675 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.651676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.651676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.651796 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.651885 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.652078 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.652150 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:39.654345 23220 catalog_manager.cc:1347] Generated new cluster ID: 654c2d8276644a7b892a53c415e401ad
I20260924 23:10:39.654399 23220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 23:10:39.661566 23220 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 23:10:39.662269 23220 catalog_manager.cc:1504] Loading token signing keys...
I20260924 23:10:39.665753 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } attempt: 3
Sep 24 23:10:39 dist-test-slave-ph9j krb5kdc[22950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790291439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 23:10:39.672799 23220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1: Generated new TSK 0
I20260924 23:10:39.672987 23220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 23:10:39.674288 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.666030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38367 (local address 127.0.0.1:46114)
0924 23:10:39.666156 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 23:10:39.666167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 23:10:39.666202 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 23:10:39.666474 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 23:10:39.666475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 23:10:39.666639 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 23:10:39.666680 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.666682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.667626 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.667627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.668461 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 23:10:39.668463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.669012 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.669013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 23:10:39.669083 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.669529 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 23:10:39.669534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 23:10:39.671058 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 23:10:39.673198 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.673199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.673200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.673418 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.673639 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 23:10:39.673640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 23:10:39.673641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 23:10:39.673699 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 23:10:39.673979 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 23:10:39.673980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 23:10:39.674020 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 23:10:39.674088 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 23:10:39.674184 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 23:10:39.674660 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.666287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.98.60:38367)
0924 23:10:39.666335 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.666337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.666343 (+ 6us) server_negotiation.cc:407] Connection header received
0924 23:10:39.666364 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.666364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.666373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.666418 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.666784 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.667513 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.668569 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.668889 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.668944 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.671146 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.671147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.671147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.671154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.673056 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.673480 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.673481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.673482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.673577 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.673762 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.673763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.673764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.673918 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.673985 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.674498 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.674573 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 23:10:39.675511 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.642478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.98.61:39945)
0924 23:10:39.642516 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.642517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.643494 (+ 977us) server_negotiation.cc:407] Connection header received
0924 23:10:39.643584 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.643585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.643595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.643690 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.649009 (+ 5319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.650065 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.654292 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.655052 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.655097 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.662876 (+ 7779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.662878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.662878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.662885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.664636 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.670226 (+ 5590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.670227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.670227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.670280 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.671211 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.671212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.671212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.671354 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.671748 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.675335 (+ 3587us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.675418 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:39.676095 23034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.642261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.0.98.62:39733)
0924 23:10:39.642304 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.642306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.642779 (+ 473us) server_negotiation.cc:407] Connection header received
0924 23:10:39.642973 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.642974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.642986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.643017 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.647175 (+ 4158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.648284 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.653286 (+ 5002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.653621 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.653662 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.660787 (+ 7125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 23:10:39.660788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 23:10:39.660788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 23:10:39.660795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 23:10:39.662664 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.670753 (+ 8089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.670754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.670755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.670812 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 23:10:39.674803 (+ 3991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 23:10:39.674805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 23:10:39.674805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 23:10:39.674939 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 23:10:39.675021 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.675905 (+ 884us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.676008 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:39.676206 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: Bootstrap starting.
I20260924 23:10:39.676934 23106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: Neither blocks nor log segments found. Creating new log.
I20260924 23:10:39.677227 23106 log.cc:824] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: Log is configured to *not* fsync() on all Append() calls
I20260924 23:10:39.678612 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: No bootstrap required, opened a new log
I20260924 23:10:39.680670 23106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.680797 23106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b113a23d5ef4496a2db5de4183a6f8b, State: Initialized, Role: FOLLOWER
I20260924 23:10:39.680871 23106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } }
I20260924 23:10:39.681166 23106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 23:10:39.681389 23080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3c8e6c101e54d929f23c32730df75f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b113a23d5ef4496a2db5de4183a6f8b"
I20260924 23:10:39.681501 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [term 0 FOLLOWER]: Advancing to term 1
W20260924 23:10:39.682008 23059 tablet.cc:2369] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 23:10:39.683432 23080 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3c8e6c101e54d929f23c32730df75f1 in term 1.
I20260924 23:10:39.683624 23079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [term 1 FOLLOWER]: Refusing update from remote peer d3c8e6c101e54d929f23c32730df75f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 23:10:39.684265 23201 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 23:10:39.687896 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.687929 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3c8e6c101e54d929f23c32730df75f1. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.688066 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:39.688125 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:39.688171 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:39.688280 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 23:10:39.692118 23242 catalog_manager.cc:1259] Loaded cluster ID: 654c2d8276644a7b892a53c415e401ad
I20260924 23:10:39.692209 23242 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: loading cluster ID for follower catalog manager: success
I20260924 23:10:39.693444 23242 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: acquiring CA information for follower catalog manager: success
I20260924 23:10:39.695200 23242 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b113a23d5ef4496a2db5de4183a6f8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 23:10:39.705040 23034 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.703244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.0.98.62:39733)
0924 23:10:39.703298 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.703300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.704292 (+ 992us) server_negotiation.cc:407] Connection header received
0924 23:10:39.704399 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.704400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.704412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.704443 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.704955 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50696: BlockingRecv error: recv got EOF from 127.0.0.1:50696 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 23:10:39.705983 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.703709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.98.61:39945)
0924 23:10:39.703750 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.703752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.705257 (+ 1505us) server_negotiation.cc:407] Connection header received
0924 23:10:39.705488 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.705490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.705500 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.705528 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.705930 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48292: BlockingRecv error: recv got EOF from 127.0.0.1:48292 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:10:39.706548 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.704014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.0.98.60:38367)
0924 23:10:39.704052 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.704055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.705438 (+ 1383us) server_negotiation.cc:407] Connection header received
0924 23:10:39.706132 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.706133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.706145 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.706174 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.706487 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46122: BlockingRecv error: recv got EOF from 127.0.0.1:46122 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:10:39.715823 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.712100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.0.98.61:39945)
0924 23:10:39.712149 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.712151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.712385 (+ 234us) server_negotiation.cc:407] Connection header received
0924 23:10:39.712490 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.712490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.712532 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.712575 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.712817 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.713651 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.714876 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.715178 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.715230 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.715449 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.715573 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.715622 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.715731 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.715749 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 23:10:39.719110 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.718344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46132 (local address 127.0.98.60:38367)
0924 23:10:39.718382 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.718384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.718580 (+ 196us) server_negotiation.cc:407] Connection header received
0924 23:10:39.718684 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.718685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.718702 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.718733 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.719056 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46132: BlockingRecv error: recv got EOF from 127.0.0.1:46132 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:10:39.724462 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.721296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.98.61:39945)
0924 23:10:39.721328 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.721330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.721536 (+ 206us) server_negotiation.cc:407] Connection header received
0924 23:10:39.721624 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.721624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.721633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.721660 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.721814 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.722575 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.723691 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.723982 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.724021 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.724215 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.724246 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.724273 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.724375 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.724393 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 23:10:39.727442 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.726686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.98.60:38367)
0924 23:10:39.726722 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.726724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.726942 (+ 218us) server_negotiation.cc:407] Connection header received
0924 23:10:39.727032 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.727033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.727043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.727071 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.727395 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46142: BlockingRecv error: recv got EOF from 127.0.0.1:46142 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:39.733817 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.730676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.0.98.61:39945)
0924 23:10:39.730706 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.730708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.730928 (+ 220us) server_negotiation.cc:407] Connection header received
0924 23:10:39.731018 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.731019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.731029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.731055 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.731226 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.731935 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.733020 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.733322 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.733360 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.733573 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.733605 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.733632 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.733733 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.733749 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 23:10:39.736621 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.735890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46158 (local address 127.0.98.60:38367)
0924 23:10:39.735925 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.735927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.736122 (+ 195us) server_negotiation.cc:407] Connection header received
0924 23:10:39.736226 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.736227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.736238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.736266 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.736571 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46158: BlockingRecv error: recv got EOF from 127.0.0.1:46158 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 23:10:39.744858 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.741629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.0.98.61:39945)
0924 23:10:39.741666 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.741667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.741931 (+ 264us) server_negotiation.cc:407] Connection header received
0924 23:10:39.742029 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.742030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.742039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.742066 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.742250 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.742996 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.744089 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.744391 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.744428 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.744610 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.744642 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.744668 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.744768 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.744784 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 23:10:39.747853 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.747112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.0.98.60:38367)
0924 23:10:39.747149 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.747151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.747351 (+ 200us) server_negotiation.cc:407] Connection header received
0924 23:10:39.747444 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.747445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.747456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.747484 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.747803 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46172: BlockingRecv error: recv got EOF from 127.0.0.1:46172 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 23:10:39.760537 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.756998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.0.98.61:39945)
0924 23:10:39.757038 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.757040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.757301 (+ 261us) server_negotiation.cc:407] Connection header received
0924 23:10:39.757400 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.757402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.757412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.757440 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.757609 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.758420 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.759644 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.759964 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.760005 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.760238 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.760271 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.760313 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.760437 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.760456 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 23:10:39.763562 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.762796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.0.98.60:38367)
0924 23:10:39.762836 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.762838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.763033 (+ 195us) server_negotiation.cc:407] Connection header received
0924 23:10:39.763125 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.763126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.763136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.763165 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.763508 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46188: BlockingRecv error: recv got EOF from 127.0.0.1:46188 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 23:10:39.784240 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.780852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.61:39945)
0924 23:10:39.780895 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.780897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.781211 (+ 314us) server_negotiation.cc:407] Connection header received
0924 23:10:39.781325 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.781326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.781337 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.781367 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.781560 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.782321 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.783430 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.783706 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.783752 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.783960 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.783992 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.784021 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.784127 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.784147 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 23:10:39.787405 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.786558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.98.60:38367)
0924 23:10:39.786598 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.786600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.786819 (+ 219us) server_negotiation.cc:407] Connection header received
0924 23:10:39.786921 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.786922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.786933 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.786963 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.787350 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46190: BlockingRecv error: recv got EOF from 127.0.0.1:46190 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:39.824123 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.820711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.0.98.61:39945)
0924 23:10:39.820756 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.820758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.821149 (+ 391us) server_negotiation.cc:407] Connection header received
0924 23:10:39.821186 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.821187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.821200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.821262 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.821435 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.822198 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.823308 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.823605 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.823681 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.823858 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.823893 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.823944 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.824019 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.824040 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 23:10:39.826895 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.825949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.0.98.60:38367)
0924 23:10:39.825996 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.825998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.826359 (+ 361us) server_negotiation.cc:407] Connection header received
0924 23:10:39.826394 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.826395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.826408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.826470 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.826819 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46198: BlockingRecv error: recv got EOF from 127.0.0.1:46198 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 23:10:39.894596 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:39.892398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.0.98.61:39945)
0924 23:10:39.892490 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.892492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.892534 (+ 42us) server_negotiation.cc:407] Connection header received
0924 23:10:39.892582 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.892583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.892593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.892658 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:39.892765 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.893238 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.894015 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:39.894191 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:39.894241 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:39.894359 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:39.894384 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:39.894421 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:39.894473 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 23:10:39.894487 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 23:10:39.896831 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:39.896326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.0.98.60:38367)
0924 23:10:39.896369 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:39.896371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:39.896394 (+ 23us) server_negotiation.cc:407] Connection header received
0924 23:10:39.896433 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:39.896434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:39.896443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:39.896483 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:39.896769 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46210: BlockingRecv error: recv got EOF from 127.0.0.1:46210 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 23:10:40.028198 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.025988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.61:39945)
0924 23:10:40.026085 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.026087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.026176 (+ 89us) server_negotiation.cc:407] Connection header received
0924 23:10:40.026203 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.026204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.026214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.026255 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.026377 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.026856 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.027563 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.027750 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.027813 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.027938 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.027964 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.028002 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.028056 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.028070 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 23:10:40.030623 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:40.029942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.98.60:38367)
0924 23:10:40.030001 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.030004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.030081 (+ 77us) server_negotiation.cc:407] Connection header received
0924 23:10:40.030193 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.030194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.030206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.030252 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:40.030560 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46224: BlockingRecv error: recv got EOF from 127.0.0.1:46224 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 23:10:40.186611 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3c8e6c101e54d929f23c32730df75f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3c8e6c101e54d929f23c32730df75f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39733 } } peers { permanent_uuid: "8b113a23d5ef4496a2db5de4183a6f8b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39945 } } peers { permanent_uuid: "5e4017755acc4aeab2a772af86a02f97" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38367 } } }
I20260924 23:10:40.186810 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3c8e6c101e54d929f23c32730df75f1 [sys.catalog]: This master's current role is: LEADER
I20260924 23:10:40.290068 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.287837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.0.98.61:39945)
0924 23:10:40.287906 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.287908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.287933 (+ 25us) server_negotiation.cc:407] Connection header received
0924 23:10:40.287958 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.287959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.287969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.288009 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.288138 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.288608 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.289412 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.289599 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.289679 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.289817 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.289844 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.289908 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.289992 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.290009 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 23:10:40.292440 23185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 23:10:40.291868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46230 (local address 127.0.98.60:38367)
0924 23:10:40.291938 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.291940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.291949 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:40.291980 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.291981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.291991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.292037 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 23:10:40.292365 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46230: BlockingRecv error: recv got EOF from 127.0.0.1:46230 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 23:10:40.618649 23198 catalog_manager.cc:1259] Loaded cluster ID: 654c2d8276644a7b892a53c415e401ad
I20260924 23:10:40.618722 23198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: loading cluster ID for follower catalog manager: success
I20260924 23:10:40.619477 23198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: acquiring CA information for follower catalog manager: success
I20260924 23:10:40.619727 23198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e4017755acc4aeab2a772af86a02f97: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 23:10:40.807746 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.805534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.0.98.61:39945)
0924 23:10:40.805701 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.805703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.805713 (+ 10us) server_negotiation.cc:407] Connection header received
0924 23:10:40.805776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.805777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.805787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.805845 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.805985 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.806476 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.807186 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.807382 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.807433 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.807547 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.807574 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.807612 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.807662 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.807680 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260924 23:10:40.811637 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.809370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.0.98.60:38367)
0924 23:10:40.809473 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.809474 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.809519 (+ 45us) server_negotiation.cc:407] Connection header received
0924 23:10:40.809573 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.809573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.809621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.809690 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.809776 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.810299 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.810962 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.811141 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.811189 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.811305 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.811406 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.811463 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.811496 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.811514 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
I20260924 23:10:40.816488 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.813303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.98.62:39733)
0924 23:10:40.813414 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.813416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.813425 (+ 9us) server_negotiation.cc:407] Connection header received
0924 23:10:40.813490 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.813491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.813543 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.813637 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.813761 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.814448 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.815406 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.815693 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.815960 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.816037 (+ 77us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.816148 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.816363 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.816383 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.816405 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260924 23:10:40.816668 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.813412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.0.98.61:39945)
0924 23:10:40.813465 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.813467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.813634 (+ 167us) server_negotiation.cc:407] Connection header received
0924 23:10:40.813721 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.813722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.813735 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.813787 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.813925 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.814677 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.815746 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.816025 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.816061 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.816257 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.816287 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.816342 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.816558 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.816584 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 23:10:40.817260 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 23:10:40.813697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.0.98.60:38367)
0924 23:10:40.813731 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 23:10:40.813733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 23:10:40.813946 (+ 213us) server_negotiation.cc:407] Connection header received
0924 23:10:40.814000 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 23:10:40.814001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 23:10:40.814010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 23:10:40.814058 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 23:10:40.814198 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.814850 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.815853 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 23:10:40.816135 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 23:10:40.816182 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 23:10:40.816428 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 23:10:40.816461 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 23:10:40.816491 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 23:10:40.817161 (+ 670us) server_negotiation.cc:299] Negotiation successful
0924 23:10:40.817184 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 23:10:40.818776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 22963
I20260924 23:10:40.824169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 23038
I20260924 23:10:40.828943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKo6W1P/build/release/bin/kudu with pid 23113
2026-09-24T23:10:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2830 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5616 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271389 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST